Nonsesne From a DevOps Engineer

The Modern Prometheus

[eztheog]

LONG TIME

Yo, so, look. I get it. You’re looking at my “blog” and thinking 

“uhhhh......what?”

SO - here are some updates of what I’ve been working on:

Bind Server at Home:

Well, I have a lot of little servers doing things at home:

4 raspberry pis

VMs - one for Bind, one for Jenkins automated builds, one for Asterisk

FreeNas server

Etc.

and I got sick and tired of memorizing IPs. SO, I built a Bind9 server and run all the DNS stuff. I also run an NGINX server for running redirection (also, I’ll be adding TLS on my stuff). I followed this blog post: https://www.digitalocean.com/community/tutorials/how-to-configure-bind-as-a-private-network-dns-server-on-ubuntu-14-04

Moved to Ansible

I stopped using Chef for my home stuff. You can see what my cookbook structure looks like here:

https://github.com/EZtheOG/cb-base-example

While I haven’t made my Ansible Playbook public (or made a bare-bones version of it), I want to make some notes as to WHY I chose Ansible.

First and foremost, SSH Keys. SO EASY.

Second, I have a bunch of little raspberry pis that act as web servers or what have you. 

THE PROBLEM is that compiling RUBY on Raspberry Pi takes 3+ hours.

So, I went to Ansible. Currently, I am still in the process of migrating my cookbook stuff and still tweaking my Raspberry-pi-specific logic out in my playbook. 

COMPUTER DIED

omg my computer died. Like, MOBO DIED. I built my machine in 2012 so ... I mean ... it was time.

SO, I built my NEW computer. REF: https://pcpartpicker.com/list/Kf3TnH

TL;DR - it’s nuts. I put in a M.2 SSD in there.  My computer is so fast that it boots into windows before DHCP sends an IP address.

HOME AUTOMATION

Along with all of these - I have been working with bridging my annoyingly hybrid home-automation hub (SmartThings) with Alexa. Honestly? It’s kind of annoying that there isn’t a more ... seamless process for this. The newest addition to my automation:

ACs are Automated with Sensibo

Time-based Automation. 

Lights turn on and music plays at a specific time every day. Just to decompress and auto-turn on lights when it gets dark out.

There will be more but ... for now that is it.

THIS HAS BEEN MY UPDATE. I’ll share my ansible project soon.

[eztheog]

THANKSGIVING QUICKIE

I’ve been cooking all day and I’m tired. So tired that I don’t want to tech.

BUT, this is what I have been up to:

https://imgur.com/a/wdzQU

I got a freenas box running and I’m creating my own little work environment at home.

4 Raspberry Pis:

1 - running Home Bridge and HomeAssistant

2 - NGINX WebServer to Proxy to each web service

3 - FreePBX, yeaaaaah got it on a Pi

4 - A Test box ... for now.

On the FreeNAS box, I got some stuff going but I have an Ubuntu VM that does the following:

1 - Runs DNS Server for my home (makes logging into my plex jail easier) 

2 - Runs OpenVPN Tunnel to my AWS Account (Starting to run apps in Docker/ECS)

3 - Jenkins - for automating all of these things and pushing them to the appropriate places

4 - Ansible Tower. I need to manage all these devices and bootstrapping them individually is annoying. Compiling Ruby on a Pi takes forever and F*** that. So, Ansible it is.

More as I get closer to tying it all together.

HAPPY THANKSGIVING EVERYONE!

[eztheog]

2016′s Final Pi Project

THIS IS LATE, I KNOW! STOP REMINDING ME

I have been working on this “ghetto” home-automation thing with my raspberry pi for awhile and I haven’t written about it because.... well.... it was janky. I am going to write a little bit of how I got to where I am _now_ but, as an overview, I created a cheap way to control outlets in my home. Super simple, I know, but it was a bit helpful. I went from manually turning on and off with a pin code, then I installed a pre-made PHP web interface, then I had a python script to automate some of it, and now I have it hooked up to Siri.  It all started this summer; it was hot and I have this old shitty AC unit in my room. I mean, like prob 1980s old. It’s through the wall of my apartment and, somehow, I signed some thing when I first moved in that makes me fully responsible for these units. Ick. Now, it’s so old that the knob was broken when I moved in, and then the stem broke off a year or two after that, so the only way to manage this crappy AC unit was to plug/unplug the unit physically. Now, what?

Well, I bought this Temperature sensor:

https://ha.privateeyepi.com/store/index.php?route=product/product&path=66&product_id=115 Which connects via the GPIO pins on the Raspberry Pi.

Then, I hooked up my breadboard and got an RFID Transmitter and Receiver. Pretty much I copied this how-to:

https://timleland.com/wireless-power-outlets/

So, I was broke and for $35 you can get 5 RFID outlet controllers as opposed to $30 for ONE WeMo remote. TBH - as I have progressed I realized the errors of my ways. Ultimately, the end goal was to control the AC via the outlet (since the AC was stuck on cause the knob broke) and turn it on/off based on temperature of the room. So, with a little python code this is what I did: https://github.com/EZtheOG/rpi-rf-outlets/blob/master/pioutlet.py The PrivateEyePi application is a python webapp that sends info to your “dashboard”.  This Python app does the following:

1. Set the desired temp to when you want to turn on or off AC

2. Scrape the WebPage to get the Temp (couldn’t figure out how to get it from device)

3. Cron job to run ever 5 min

It’s a bit ugly, for sure but it worked. And that’s all that matters!

[eztheog]

Ooof... Writing about CloudFormation

WOAAAAH!  It’s been awhile since I posted. It’s so funny, too. I keep telling myself “let’s write some stuff that you’ve done” (again, to myself in my head) and then I just have so many projects so I don’t know what to write about.

So, if you’re in DevOps YOU SHOULD KNOW about CloudFormation. It’s about managing your services and resources in “stacks”. If you don’t know about this - read up on it. Your life will be better.

My company uses SparkelFormation (sparkelformation.io). It’s like Terraform ... but Ruby based. I mean, it’s ok - I just inherited it when I got hired.

But, when you manage all your resources and stacks as one - it reduces complexity. Now, granted, this is prob 1 step behind everyone else with their Dockers and Kubernetes, but that’s it.

SHORT AND SWEET

[eztheog]

Writing Chef Cookbooks

So, here’s a question. HOW do you write your cookbooks? What is the best way to do it? Do you make your code clean by putting everything into the attributes, or do you list everything you are doing in your run list? I have been thinking about this for quite some time.  I am migrating from Masterless Puppet to Chef. There’s a lot of reasons but the masterless puppet is a bit over engineered and I want to reduce the code footprint on our nodes.  BUT, I have been writing a lot of stuff in Puppet with Hiera, which is an awesome way to obfuscate all of your sensitive materials and just have the code. So, I liked that idea in Chef. I STARTED WRITING IT. I made all of my chef code to be very Puppet-esque in this manner. I just wanted to write clean code. So, in my attributes I did this:

 [default['system']['pkg'] =  [ 'amqp','traceroute','tcpdump','binutils','git','rsync','ruby','bash','curl','gcc']

and then in my recipe default.rb (e.g.) I just do something simple like :

node['system']['pkg'].each do |pkg|    package pkg do        action :install    end end

Looks great right? Just nice and clean. Well, there IS a problem with this way of doing it. I got in a conversation with coderanger on the #CHEF IRC channel about this ideology and he was telling me that it isn’t the best way to do it. His reasoning is that you are reading code more than you are writing code, so the more you obfuscate data then the less readable your recipes are. 

Well, UGH NOW WHAT? I just wrote all this code to be cool and clean and this guy (WHOM I DO ADMIRE) says it’s not the best way.  So, I wrote cookbooks the Chef-ly documented way. But, it got me to thinking - what is the RIGHT WAY? I called a friend, another brilliant person, Nathan Milford, and talked to him about it. After talking to him about my dilemma I kind of came to this point - each thing you do at one job is NOT directly repeatable to your next job. 

So, what do you do? Well, in my opinion, whichever makes more sense. For smaller environments - this probably isn’t a terrible idea -- especially for a base cookbook where things don’t necessarily change too often. But, in a bigger place where you utilize cookbooks for your entire deployment procedure, I’d probably think of something else.

Anyways, that’s my two cents on the matter.

[eztheog]

PACKER AND CHEF

I haven’t written much cause I haven’t done anything cool or different. Just day-to-day nonsense.

BUT, one thing I have had fun with recently is using Packer. I have Packer creating an instance, bootstrapping with Chef, and then creating an AMI. This is specifically perfect for my company’s game environment in Elastic Beanstalk. Added bonus? Create a Jenkins job to automate it.

Packer has a TON of stuff that it can do - and what I am going to show is very very very basic. But, my setup is this : A Packer machine in AWS on a T2.Micro, a Base Chef Cookbook, a recipe for AMIs, and a JSON template for my Region to create the AMI.

For starters, my JSON looks like this:

{ "variables": { "aws_access_key": "ACCESS_KEY", "aws_secret_key": "SECRET_KEY" }, "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-west-2", "source_ami": "ami-d89d49b8", "security_group_id": "sg-0gj43776b", "subnet_id": "subnet-fsa3a998", "instance_type": "t2.micro", "ssh_username": "ec2-user", "iam_instance_profile": "IAM_PROFILE", "ami_name": " COMPANY-Beanstalk-HVM {{timestamp}}" }], "provisioners": [ { "type": "shell", "inline": [ "sleep 30", "sudo yum update -y", "sudo mkdir -p /etc/chef/ohai/hints", "sudo touch /etc/chef/ohai/hints/ec2.json", "sudo mkdir /tmp/packer-chef-client", "sudo AWS_ACCESS_KEY_ID={{user `aws_access_key`}} AWS_SECRET_ACCESS_KEY={{user `aws_secret_key`}} aws s3 sync s3://S3Bucket/chef/ /tmp/packer-chef-client/", "sudo AWS_ACCESS_KEY_ID={{user `aws_access_key`}} AWS_SECRET_ACCESS_KEY={{user `aws_secret_key`}} aws s3 sync s3://S3Bucket/chef/ /etc/chef/", "sudo chmod -R 777 /etc/chef/", "sudo chown -R ec2-user:ec2-user /tmp/packer-chef-client", "sudo chmod -R 777 /tmp/packer-chef-client", "sudo mkdir -p /var/log/chef" ] }, { "type": "chef-client", "server_url": "https://chef.mycompany.com/organizations/myorg", "run_list": ["cb-base","cb-base::ami_bootstrap"], "ssl_verify_mode": "verify_none", "staging_directory": "/tmp/packer-chef-client/", "config_template": "/tmp/packer-chef-client/client.rb", "skip_clean_node": true, "skip_clean_client": true, "validation_key_path": "/etc/chef/validator.pem" }, { "type": "shell", "inline": [ "sudo rm -rf /etc/chef/client.pem", "sudo /etc/init.d/sensu-client stop", "sudo /etc/init.d/chef-client stop" ] }] } `</pre> Now, note - this code is ugly. I am just kind of done and figured out and you're getting the "before I clean up and make it nice" code. Regardless, it works. Moving on... I have an AMI recipe that does the following: I create a /.bootstrap directory (so it's "hidden") and in there I have a small script to set hostname on initial boot. Why? so the initial chef run does not fail. Now, for my beanstalk env environment - it's easy. All the names of an application are "my-application" and then I just add instance_id to them. looks like this: <pre>`#!/bin/bash NAME="$(python /.bootstrap/get-env.py)" ENV=$(echo $GET_ENV | awk -F'-' '{print $2}') FIND_INST="$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)" INST_ID=${FIND_INST:2} REGION='use1' HOST=" $GET_ENV-$INST_ID.$REGION.mycompany.com" hostname $HOST chef-client -r role[cookbook-$ENV] `</pre> and then I have an added line in /etc/rc.local-> `/usr/bin/firstboot` and in /usr/bin/firstboot: <pre>`#!/bin/bash FLAG="/var/log/firstboot.log" if [ ! -f $FLAG ]; then sudo /.bootstrap/bootstrap.sh /bin/touch $FLAG fi

The Elastic Beanstalk App name is "My-Application" and the HOSTNAME will be set as: my-application-INSTANCEID.mycompany.com

My AMI is created with Packer - it runs the Chef Base cookbook I have created, adds files for firstboot of new machines to set hostname and just to run one extra cookbook for my elastic beanstalk application environment.

Something small and easy to do. This has had HUGE positive returns in my infrastructure - in that it takes now 30 seconds to swap out instances and get the AWS healthcheck to pass. Whereas, before, it took about 4-5 minutes per instance.

[eztheog]

Notes about IPSec.D and AWS

It’s been awhile since I wrote. My boss quit and I’ve been by myself. I have found this to be something that happens to me a lot; person who hires me quits after a couple of months, now I am there to hold the team together(?). What is funny is that the team is just me.

A couple of months ago, probably 4, I wrote about IPSec.d and MTU clamping and the problem I had about MTU negotiation between multiple AWS Regions. Don’t want to scroll? HERE. 

First, my opinion on this is AWS should really support region-to-region VPNs within their product. The fact that you do this yourself by either (a) doing it yourself or (b) using the AWS Marketplace is stupid. Really stupid. I hate it. It is such an oversight on their part.

Second, the documentation on how to set up IPSec.D is pretty straightforward but only applies to the application (openswan, strongswan) and how to connect the VPN tunnel. But, there are some things that do not exist in that documentation that I would like to point out. 

Now, I am learning all of this with no guidance so to some of you readers will go “duh”. I mean, I said “duh” after I banged my head against a brick wall but it is something I overlooked.

The first and important thing is to use DNS names for your endpoints. This will make changing VPN nodes down the road easier.

Routes do NOT get automagically populated. This is something I could not figure out cause I took AWS for granted. Routes need to be managed in the VPC and point them.

TL;DR - VPC routes need to be assigned to the VPN node that is establishing the connection.

Add the IPs to the Routes (in the Route Tables) and the Target should be the NIC assigned to the VPN node.

Check that the routes got populated from other nodes in the same region.

Look - you are probably reading this going DUH. I got so safe with AWS magically handling everything that this was a dumb venture for me.

I am bringing awareness to dumbness.

UNTIL NEXT TIME!

[eztheog]

Puppet to Chef

OKKKKKKKKKKKKKK - long time I posted but I got some cool projects and other things to write about but they aren’t done. Some really cool Raspberry Pi project I am working on and I am getting there. But, I have some stuff that’s on my mind and I want to talk about it. I work for a mobile gaming studio in NYC. We are fully AWS, as I suspect most people are, and we try to leverage automation as much as possible. Granted, we’re not 100% there, but we’re getting there. 

So, at work we are using Masterless Puppet for all of our nodes. The structure is fairly simple, have all manifests in one repo, hiera obfuscates all the variables, and each node downloads the same manifest across the infrastructure. Now, the way it was done was that each node gets bootstrapped with a bash script via User-Data. Now, the bootstrap sets up : 

hostname some yum repo stuff git clone of the puppet masterless manifest a cron job to run puppet every x minutes So, at initial bootstrap, the puppet run takes about 5 mins + (depending on instance type) as it goes through the entire manifest. Now, our infrastructure is small enough that this isn’t a huge problem but it will be at scale. I HAVE DECIDED TO GO BACK TO CHEF.

For a couple of reasons but the main two are to segment the roles a bit so that not each node are going to get code for everything and that the initial bootstrap of our nodes are quicker. 5 mins for deployment is a bit too long. Sure, you can use AMIs or Packer but that adds a level of administrative headache that I don’t have time to maintain at the moment. (Though, Packer is on my list of things to do). Original idea? Have nodes bootstrap themselves, even our application, at launch. AMIs become outdated and this would allow us to maintain our nodes to be up-to-date with packages. Now, before you go gaga, this idea will mature with Packer and some other glue that I have yet to fully flush.  Well, that’s it for now ... but ... more later! 

[eztheog]

AWS, IPSec.d VPN, ElastiCache, and the Terrible, No Good, MTU Negotiation

WHAT A MOUTHFUL.

MY GOD -- I just fixed this and I am going to write this down NOW. It’s fresh in my memory. I am going to describe my situation, explain how to fix it, and hopefully you can understand if you are looking for an answer. This was super annoying. I was so mad. I was like Bernie Mac:

I am in AWS (DUUUUHHHH) and I am using multiple regions for inherited reasons. Just as an example, let’s just say that I am in Singapore (ap-southeast-1) and in Sydney Australia (ap-southeast-2). Now, all of my main tools, EastiCache DBs, and apps are in Singapore and I am creating a secondary region for Sydney for new apps, and some newer tools. Now, you need access. So? VPN. Sure, you could do Hardware VPN (which is probably easier, as I shrug) but the free way is to do an opensource VPN solution.

Now, I am not going to write HOW to setup VPN. Because,** Raymii** and Alex @zeitgeist​ have great in-depth ways of setting this up. Just have the two VPN servers, one in each region, have a Public IP and point them to each other. Make sure your Security Groups allow for the UDP rules from each other and that’s it.

If your ElastiCache, or any service, is required across the VPN tunnel, then there are some caveats when connecting. Specifically, though, for ElastiCache. Strongswann IPSec.d is JUST the Tunnel. Maybe that is apparent to you, reader, but it wasn’t to me. The Routes are still regulated on the AWS level, so you need to go to VPC in each region and make sure you populate those routes on the correct VPC. Strongswann just connects, but any servers downstream do not know the routes directly.

Another challenge is MTU negotiation. Even if you let ICMP open on all SGs (Security Groups) for all internal IPs, the MTU is still not auto-negotiated on the VPN tunnel. It’s kind of a pain in the neck. I have been banging my head against the proverbial wall for a couple of weeks here, opened some AWS tickets, and got to a solution.

The problems to note:

ElastiCache **BLOCKS **ICMP by default and cannot be overwritten (even if you change the SG)

StrongSwann does not negotiate MTU by default either. Security Groups can be set, but sometimes those MTU blocks are dropped and connection from VPN1 to VPN2 can die..without much warning.

In my specific situation, I needed to access our ElastiCache cluster over the wire for some read reasons. Now, the MTU would drop and I had to set MTU manually. THAT IS ANNOYING. I don’t want to do that for every server that needs to make a call across the wire.

So, with the help of my friends - AWS - I found that all I needed to do this on my VPN nodes:

sudo iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu_ sudo iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128

I am running configuration with Puppet, ick, but if you use the PuppetLabs Firewall module, here, you can easily set your rules by either:

firewall { '110 TCPMSS for VPN clients': chain 'FORWARD', table 'mangle', source '10.0.2.0/24', proto tcp, tcp_flags 'SYN,RST SYN', mss '1361:1541', set_mss '1360', jump 'TCPMSS', }

OR - if you use hiera:

#firewall.pp class vpn::iptables inherits vpn::params { create_resources(’firewall’,$iptables) } #params.pp class vpn::params { $iptables = hiera(’vpn::iptable::rules, {}’) } #hiera where rule is located --- vpn::iptable::rules: '1-TCPMSS-for-VPN': chain: 'FORWARD' proto: 'tcp' tcp_flags: "SYN,RST SYN" jump: "TCPMSS" clamp_mss_to_pmtu: true ensure: 'present' '2-TCPMSS-for-VPN': chain: 'FORWARD' proto: 'tcp' tcp_flags: "SYN,RST SYN" jump: "TCPMSS" set_mss: "128" ensure: "present"

Now, you can test your connection and that should be it.

I’m adding this info so that it’s on the internet. I saw a lot of people saying that this was an issue but it wasn’t apparent that there was a ready-available solution.

[eztheog]

Self-signed Certs, Jenkins, and NGINX.

Self-signed certs is something I have written about, mostly on how to set it up. The original need for self-signed certs was that we wanted to securely lock-down internet-facing applications. Self-signed certificate authority worked because I could secure external access by putting applications behind NGINX and make it so that you had to have a SSL cert signed by the same authority. We didn’t have money for a REAL VeriSign cert and this was a perfect solution for the small tech-startup.

via GIPHY

Listen, OpenLDAP is cool but it creates a level of administrative work that, in my case, I did not have time for. To be honest, it’s easier to make one cert per person (in fact, I scripted it), and for the Dev and Business-side person (e.g. BizDev, Product, TAM teams) it’s just easier to let them work without managing passwords, resetting, etc. There is a cost-benefit analysis that needs to be thought out. Self-signed certs is a great medium to manage security with ease-of-access for those who have been granted access.

Anyways, so I ran into a problem by protecting our Jenkins box. We use Jenkins for everything, and while it’s ‘secure’ with local usernames and passwords, it’s just not good enough. So, I put Jenkins behind NGINX and our certs into a directory:

sudo mkdir /etc/nginx/ssl

move certs there.

I will paste my conf file and explain the parts:

upstream jenkins { server 127.0.0.1:8080 fail_timeout=0; } server { listen 80; server_name jenkins.company.com; location / { return 301 https://$server_name$request_uri; } location /github-webhook/ { proxy_pass http://jenkins; } } server { listen 443; server_name jenkins.company.com; root /var/lib/jenkins/jenkins.war; access_log /var/log/nginx/jenkins.access_log main; error_log /var/log/nginx/jenkins.error_log info; ssl on; ssl_certificate /etc/nginx/ssl/jenkins.crt; ssl_certificate_key /etc/nginx/ssl/jenkins.key; ssl_client_certificate /etc/nginx/ssl/root-chain.crt; ssl_verify_client on; ssl_verify_depth 5; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect http:// https://; proxy_pass http://jenkins; } }

Here is what this config does (for those who don't know) and why it does it:

First, we put the jenkins box behind NGINX and from the outside block port 8080. Then, we rewrite all top-level access (http://jenkins.company.com) to https:// as a 301 permanent route. But, we allow access to jenkins.company.com/github-hook/ because, well, it would deny all Jenkins jobs to access Github to run after a commit has been push. I made this mistake, as I wasn't thinking through, and this is how I was able to maintain that feature.

Under server, we just have it run on 443. Point servername and root and access log to whatever your jenkins and naming conventions are.

Under ssl Cert, we map the crt and key files to the ../ssl directory and we set ssl_verify_client on as that is what checks the validation and the depth to be 5. For self-signed certs, you have to make sure NGINX checks the root-chain cert for Immediate and Root Authority by adding depth on here.

The rest? ehh some Proxy stuff.

I decided to post this up as I found a lot of people ASKING how to do this but no one actually posting an answer. So, hope this helps!

via GIPHY

[eztheog]

The Practicality of Sensu

If Jan Brady were a Linux SysOps, she would’ve said ‘Monitoring, Monitoring, Monitoring’.

Everyone is about monitoring. EVERYONE. There are a ton of great articles on the topic, my favorite being from Ian Unruh from his Monitor Everything posts which gives a good idea how to structure your data, centralize your logging, and level the intensity of the logging across your infrastructure.

From my experience as a DevOps engineer, I have used Sensu, Graphite (with Grafana), PagerDuty, DataDog, SumoLogic, and some other collectors like CollectD and StatsD. This experience covers logging, monitoring, alerting, and the actions of alerting.

Today, though, I am going to talk about Sensu but in a very practical way. A lot of this is going to be a bit of a how-to, not so much how to install, but how Sensu functions and how to work with it. Their documentation is great, but I always felt that I have a very specific question in regards to their syntax and that it wasn’t documented. Now, Sensu is great, I hang out in their IRC Channel and the people in there are very very helpful. Sean Porter (@portertech), one who created Sensu, has answered questions directly. They are all great. But, that does not mean that some things can go unsaid. So, that’s what this post is about.

For those of you who do not know, I spent 7 years in IT and I am somewhat new to the Linux SysAdmin/DevOps world. I love it. IF you are in IT, there are a lot of Linux solutions that can translate over and I think Sensu is one of them. But, before we get started here are somethings to note:

I am not going to go into how to install Sensu. Not only did I cover this on Centos7 but, realistically, Sensu’s documentation on installation is amazing. The team keeps their Chef Cookbooks and Puppet Manifests up-to-date regularly.

These talking points are just that, talking points. I will give you my best ideas and opinions. I am new to a lot of this so my ideas may be good, but might be inaccurate. My point is for you to learn from my ignorance. ;-)

And that you’ve at least perused the documentation. Listen, I’ve only read it because I’ve skimmed it enough times. ;-). I also work for a team that believes in RTFM. So, there’s that.

Structured Data from the Beginning

From the start, when you are deploying your Sensu server and clients, you need to make sure that you come up with a logical schema for which checks are run on different machines. Think of subscriptions as groups, you want a ‘common’ group, and an $application group. For example, you will want to have a bunch of basic checks that are ran on every machine: (e.g.) cpu usage, memory, disk space, if your conf. management client is running (chef/puppet/etc.), etc. But, if you have web-application servers, you will not want all the nodes to check to see if HTTPD is running. Why? Outside of the obvious answer, the cleanup for removing those checks is a pain. I was testing a check in the ‘common’ subscription group and it got pushed to every node with the ‘common’ subscription tag. Dumb.

Everyone is going to do it differently, but the naming convention for subscriptions I chose relate to the higher puppet roles/profiles that we have setting up the system. In chef, it would be based on the groups. So, every node gets a ‘common’ subscriber but then a ‘role’ subscription. So, if I have a bunch of web nodes, then those servers in that role would have a ‘webnode’ subscriber. This makes it easier to configure the checks and deploy them across all the nodes. Now, I work in a masterless puppet system, so every node gets the entire manifest. Pushing all the nodes with all the checks does no real damage because one server will only use the checks associated with it’s subscription.

Now, you can change the naming convention (and it will be easier to change using a CM) but getting most of the things separated at the beginning is easier than doing it later. Trust me.

The other way that I keep my data structured, is I create subdirectories in /etc/sensu/conf.d/ for each different part of the application whether it be checks, handlers, or other tasks. In /etc/sensu the default is that the subfolders handlers, which have the action-based scripts, and plugins, which have all the ruby/perl/bash that are used for the checks themselves. In my configuration, the /etc/sensu/conf.d/ is where the checks folder has all of the defined checks for the system; where the handler folder has the json formats for what do do with the upper-level handlers, and some other subfolders. For example, I have an ‘alert’ folder, which has all of the json files for alerting-based mechanisms. So, that is where I have my hipchat json defined, my email, my sms, etc.

Personally, I don’t like a subfolder with a mishmash of config files. I want to keep my infrastructure clean so that anyone can come on board and logically peruse the file tree and figure out where things are if they need to make a temporary manually a check.

Understanding and Testing Your Checks

Your check_yourchecking.json will need to go in two places for the check to work, the local machine you are checking, and the server so that the check registers with the server. I know, I know, it seems pretty sensible when you read it but it wasn’t apparent when I was going through this method.

When you’re testing your json formats for the checks, I always use an online JSON checker. This always helps me when I miss a {} or a comma. This saves a ton of time. Again, seems logical but, again, something that helped me out when I was doing this.

But, when you have your check ready, always test it. DUHHHHHH.

I always write the check to a dummy machine (test.mycompany.com) and I run the test in the subscriber group “TEST” or something dumb.

It will be easier before you use your CM to deploy it.

Get Your Rhythm 

Find your start point, for me? Well, I started with system and processes checks. Something basic and something you can easily test - well, mostly if a process is or is not running.

It was easy to generate the quick results so that when you’re building your system out it’s easy to test.

OK

Well, I have been writing this out for so long BUT I hope this helps all you beginners out there.

I am always asking questions and I am always learning. Trust me, I’ve asked some dumb questions and still do.

Thanks for reading! 

[eztheog]

Installing Sensu on CentOS 7

Normally, I’m an Ubuntu dude for all of my servers and messings-arounds at home, but at work I feel like I’m always in a CentOS environment. Not hating, just saying how it is.

So, I got this new job and they use some technology I don’t know, one being Puppet. I’m a Chef Guy, by institutional inheritance and no other reason. So, while I’m trying to figure out how Puppet works, I figured I wanted to get my Sensu skills up.

I finally jumped onto the CentOS 7 world - skipping 6 entirely - and this is what I found ..... maaaan this is a whole new world as nothing seems to work and I’ve had to jimmy rig repos to work. Well, DON’T Choose minimal install but if you did - follow me:

* note * Sensu website has amazing documentation and I have merely copied from them. I AM NOT A GENIUS. Seriously, Sensu is already awesome.

https://sensuapp.org/docs/0.19/installation-overview

I’m a pirate of information

PREP

sudo yum install epel-release

sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

sudo rpm -Uvh http://archive.linux.duke.edu/pub/epel/7/x86_64/e/epel-release-7-2.noarch.rpm

sudo rpm --import http://packages.erlang-solutions.com/rpm/erlang_solutions.asc

sudo rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc

Install ERLANG

sudo yum install -y erlang

Install RabbitMQ

sudo rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v3.5.0/rabbitmq-server-3.5.0-1.noarch.rpm

sudo chkconfig rabbitmq-server on

sudo /etc/init.d/rabbitmq-server start

Add RabbitMQ Host for Sensu

sudo rabbitmqctl add_vhost /sensu

sudo rabbitmqctl add_user sensu $PASSWORD

sudo rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

Install Redis & Turn it on

sudo yum install -y redis

sudo chkconfig redis on

sudo systemctl start redis.service

Check Redis

sudo systemctl status redis.service

sudo redis-cli ping

INSTALL SENSU

Ok - first we gotta add sensu repo:

echo '[sensu] 87 name=sensu 88 baseurl=http://repos.sensuapp.org/yum/el/$basearch/ 89 gpgcheck=0 90 enabled=1' | sudo tee /etc/yum.repos.d/sensu.repo 91 cat /etc/yum.repos.d/sensu.repo

then

sudo yum install -y sensu

make sensu owner of /etc/sensu:

sudo chown -R sensu:sensu /etc/sensu

Download some JSON files for configuration templates:

Connections:

sudo wget -O /etc/sensu/config.json http://sensuapp.org/docs/0.19/files/config.json

Checks:

sudo wget -O /etc/sensu/conf.d/check_memory.json http://sensuapp.org/docs/0.19/files/check_memory.json

Default Handler:

sudo wget -O /etc/sensu/conf.d/default_handler.json http://sensuapp.org/docs/0.19/files/default_handler.json

Start Sensu:

sudo /etc/init.d/sensu-api start

sudo /etc/init.d/sensu-server start

LAST PART - Install the Dashboard and Start it:

sudo yum install -y uchiwa

sudo /etc/init.d/uchiwa start

And that’s it! Just hit your IP at http://IPADDRESS:3000

and you can hit it via dashboard.

But you have to configure checks.

WANT TO READ AN WESOME ARTICLE? Ian Unruh has a great article about how to monitor everything. 

https://ianunruh.com/2014/05/monitor-everything-part-4.html

[eztheog]

My laziness

Hey friends, I've been away for a bit but let me give you a brief update: I'm not an IT Manager anymore, I'm a DevOps engineer! My posts will be more devops related in nature moving forward. Also, I also write for tech radar doing product reviews. I love it. Check out some of my articles here: http://www.techradar.com/us/author/ez-bardeguez I've got some cool chef AWS automation posts to finish up so stay tuned!

[eztheog]

FreePBX - Critical Zero-Day Vulnerability

I follow NerdVittles for most Asterisk/FreePBX based needs.  It's a blog that covers reviews on tablets, android, etc. but, at its core, focuses really on Asterisk phone systems.

It's great!  I've been following it for the past 3-4 years now and the blog always has some cool new feature that I can add on my FreePBX machine at home (I run it on a raspberry pi) such as How to add a Security system to your Asterisk box.

Anyways, about 2 weeks ago, FreePBX.org and NerdVittles wrote articles about a huge vulnerability that affects ALL Asterisk machines running FreePBX on ALL linux flavors.  

What happens?

Well, a vulnerability was found in the Asterisk ARI Framework Module that allows users to bypass any user/pass authentication which can grant a remote hacker full remote execution access through Apache.

Similarly to Heartbleed, this is a vulnerability that has been around a very long time and people just noticed it now.  All versions of FreePBX prior to 12 as directly affected and FreePBX 12 is still vulnerable if you keep the legacy ARI module enabled and installed. 

How to Fix It?

Sourcing information from FreePBX (http://www.freepbx.org/node/92822) - the way to patch your files are as follows:

Users prior to FreePBX 12 should update FreePBX ARI Framework to version 2.11.1.5 immediately

FreePBX 12 users should disable and uninstall the legacy FreePBX ARI Framework module and switch to the new User Control Panel, which is not to be confused with the previous ‘User Control Panel Tab’. 

Read http://www.freepbx.org/node/92822 for more information on exactly what to do.

NerdVittles also has a good explanation on what exactly this is here.

Make sure your firewall has shut off external access to the web and ssh (if possible).  While your server will still be "vulnerable" - if it isn't accessible externally (except for SIP or IAX2 trunks) then you should be fine.

[eztheog]

AWS S3 Buckets and Hyphens

Don’t do it.

Period.

The more I work with AWS the more I realize that I kind of hate it.  Well, love/hate - but mostly hate.  It’s intuitive non-intuitive interface and setups are super annoying.  I created an S3 Bucket for a Ruby/Rails app and I called it:

"Name-Of-App" (taking name out for a big client and what the product is)

The developer kept getting this issue:

#(an email from the dev)

Also, I get this warning in my rails console: fog: the specified s3 bucket name(“Name-Of-App”) is not a valid dns name, which will negatively impact performance.  For details see:http://docs.amazonwebservices.com/AmazonS3/latest/dev/BucketRestrictions.html

So - while this is a note for me - I hope you, reader, read this and don’t get into a production issue like I did.

Good luck and DON’T MAKE THE SAME MISTAKE AS I DID!

[staff]

Today’s the day. The day you help save the internet from being ruined.

Ready? 

Yes, you are, and we’re ready to help you.

(Long story short: The FCC is about to make a critical decision as to whether or not internet service providers have to treat all traffic equally. If they choose wrong, then the internet where anyone can start a website for any reason at all, the internet that’s been so momentous, funny, weird, and surprising—that internet could cease to exist. Here’s your chance to preserve a beautiful thing.)

[eztheog]

Dell PowerConnects - heaven and hell

My resume has a super slick mission statement that is something along the lines of:

          Primary objective is to implement low cost and open source

          initiatives to lower IT overhead costs.

Yeah, it reads pretentious but there is both a lot of value and truth to this statement. I like not spending money on licensing contracts with Microsoft and making my own stuff work. I then like taking the money I save and writing in my annual review:

"I saved the company $xx,xxxx this fiscal year"

Or, I take some left over money and invest it in upgrading the company’s infrastructure.

The Dell Powerconnect was my treat in this realm.

I love these switches. I had the opportunity to work with these switches at my previous employer, Outbrain, and decided to give these a whirl.

Why Dell?  Why these? 

I love how the switches auto-stack with HDMI cables.

Voice vlan. Take the OUI of each phone you have in your environment and put them in a voice vlan database. You label the VLAN # and make it the Voice Vlan. Plug in the phone, tag it to the vlan, and boom. It auto tags the data. You can then plug your computer into your VoIP phone and it sends that data over the data vlan. Its so rad.  #oversimplifying

Web interface for the non-technical - you can do everything that CLI does.

I’m migrating from physical switch racks, one for data and the other for voice, to vlans and voice vlans. My company, MKG, is expanding floors in the building. We are on floor 4 and now floor 9. I wanted to consolidate switches but wanted a working proof-of-concept before I started spending money for the entire agency. So, I went out and bought 2 Dell PowerConnect 5548P switches off of eBay and got rolling.

/>

Now, since I still have the old physical switches segregated on the 4th floor, I had to create some undesirable switch configs.

Stacked both switches (1/0/1 and 2/0/1).

Ran 4 180ft Cat6 from 4th Floor to 9th Floor and terminated into a Patch Panel. (2 Data Cables and 2 Voice)

I plugged from Data Patch into Switch Ports 1/0/47 and 2/0/47 and Voice Patch into Switch Ports 1/0/48 and 2/0/48. (Failover?)

I then created 2 VLANS for Voice and Data — 100 for Data and 101 for Voice

configure vlan database vlan 100  exit

For the uplink Data ports they are setup as 

configure interface gigabitethernet 1/0/47 switchport access vlan [100 Data/101 Voice] exit

and Voice is the same except change VLAN #.

WHY ACCESS?!

Because, the PowerConnects are connected to dumb switches.  Yeah, I said it.

You can’t trunk the port (in this scenario) because when the data leaves the PConnects to the switches downstairs, those switches won’t know what to do with tagged frames.  The Dumb Switches will be like:

Now, for the rest of the ports on the switch that will have phones plugged into:

spanning-tree portfast

switchport mode trunk

switchport trunk native vlan 100

switchport trunk allowed vlan remove 101

voice vlan enable

Plug in your VOIP Phone and tag it on the 101 VLAN.

PROTIP - To configure a BUNCH of ports do

configure

interface range gigabitethernet 1/0/1-46

Now, have fun showing this off in your office.  You will be able to consolidate ports, have less infrastructure equipment, and can manage things from one place (one stack).

Go ahead, show your boss

FOR MORE INFORMATION

I found some good links to help you out:

One From Dell - I ignored the whole ACL permissions

Steve Jenkins’ Blog - Has a really good setup blog post BUT his CLI commands were not 100% compatible with my switch. DEFINITELY helped though.