Andrew Mussey :: Apps, Code, & Comments

The previously personal blog of Andrew Mussey: Software Developer at Samba TV, DJ Mu-Z, an independent videographer, and a Virginia Tech Hokie

[amussey]

Indenting data bag JSON and Improving diffs

During the creation of our team's encrypted data bags for Chef, the JSON was generated on a single line in the files.  While this is great for keeping the size small during deployments, it's not the most helpful when looking at a diff on github to see what was changed.

We found while we were re-encoding the data bags recently that the JSON was being split out into a more "prettified" format by a newer version of Knife.  I put together a line of code (mixed bash and Python) to navigate through a folder tree and re-format the other JSON files in the same way as the new version of Knife (with a 2 space indention instead of the Python JSON library's standard 4):

find . -type f -iname "*.json" -print0 | while IFS= read -r -d $'\0' line; do if [ $(cat "$line" | wc -l) -eq 0 ] ; then JSONBLOB=$(cat "$line" | python -c "import json, sys; print(json.dumps(json.loads(sys.stdin.read()), indent=2, separators=(',', ': ')))"); echo "$JSONBLOB" > "$line"; fi; done

Here is the code spaced out into a slightly more readable format:

find . -type f -iname "*.json" -print0 | while IFS= read -r -d $'\0' line do if [ $(cat "$line" | wc -l) -eq 0 ] then JSONBLOB=$(cat "$line" | python -c "import json, sys; print(json.dumps(json.loads(sys.stdin.read()), indent=2, separators=(',', ': ')))") echo "$JSONBLOB" > "$line" fi done

Copying and pasting this command will run this in the current directory and all subdirectories.  You can change the directory this is run on by changing the target in find command to find /directory/path.

[amussey]

Virginia Tech TV: A Website Makeover and Software Design Practices

I recently helped Virginia Tech Television (VTTV) put together their new website (soon to be available at both vttv.vt.edu and vttv33.com, currently available at vttv33.com/beta).  The previous site, a well designed custom PHP CMS, has unfortunately grown stagnant from neglect after the station stopped using Blip.tv in favor of YouTube.

This sort of neglect is an almost constant problem with student organization websites.  Those who come in and understand enough code to construct a new page are usually around just long enough to put a page online, maintain the new system for a year, and then graduate or move on to other projects.  The system sits infrequently modified from that point on, leading to outdated information and, in the case of many Wordpress installations, copious amounts of spam.

To hopefully mitigate that, thew new VTTV site was designed with two goals in mind: to give basic (easily updatable) information, and to get visitors over to the consistently-maintained YouTube channel.

(The cats are placeholders for ads.)

The Tech

The new site was designed in an afternoon on top of Bootstrap 3.  While the design is nothing terribly unique, it achieves two goals it was designed for.  Under the hood, the site uses jQuery for a small selection of dynamic content and PHP for fetching data from on-disk files.  PHP is also used to fetch the YouTube thumbnails displayed on the landing page.

A full breakdown of the site's tech can be viewed here.

Scrapping the Database

Part of the maintainability aspect that I wanted to improve with the new site was limiting the number of moving parts.  Part of that goal involved the backup and mobility of the site.  While the next webmaster might not be code savvy, basic instructions could hopefully allow for easy updates.

A big piece in this simplification lay in data storage.  While storing data in MySQL is not highly complex (and a fundamental part of the LAMP stack), it does add another piece for someone to remember as they are migrating or backing up the website.  For this reason, the site was designed to store and read information from configuration files.  These configuration files contain no private or sensitive information, so they are able to be stored almost anywhere on the server without concern.

To make these configuration files easy to edit, a simple administration panel was put together.  All dynamic content on the site is reachable through this password protected panel.

The website isn't configured on the front of vttv33.com quite yet; it should be ready in the coming weeks.  If you'd like to check it out as the finishing touches are being made, visit vttv33.com/beta.

If you'd like to view the code behind the new website, it is stored in a git repo and publicly available on BitBucket.

[amussey]

At the end of September, I took part in a Startup Weekend for the first time.  While I had participated in a number of Hackathons, Startup Weekend Blacksburg was the first competition that wasn't entirely "code code code!", but additionally focused on business development and market validation.

After pitches on the first night, I joined a team with six other people, consisting of five software developers, and two business majors.  The pitch, which later adopted the name Pet My Pets, was for a website for pet owners that would match people who needed their pets watched with people who wanted pets but for one reason couldn't take one on full-time.  Effectively, it was AirBnb for pets.

From left to right: Joe Fletcher, Matthew Hubbard, Michael Liu, Wilson Mitchell, Subir Jolly, Phil Sayre, and Andrew Mussey.

Our focus for the weekend was to build the website and to verify that there were customers that would be willing to either have their pets watched or to "rent" pets.  Interviews with people throughout the Blacksburg community resulted an overwhelmingly positive response on both sides.  While some people were concerned about letting a random person from the internet watch their pet, they said they would be put at ease if there was some sort of verification or background checking in place.

For our website, we set up a Python Django application backed by SQL Lite with a Bootstrap template on top.  SQL Lite functioned perfectly for rapid, distributed development.  If the idea were to progress beyond Startup Weekend, we had planned to migrate to MySQL.  Dynamic content on the website was scripted using jQuery, and the page was hosted on AWS (thanks to the free Startup Weekend credits).  While the app isn't online yet, we were able to launch a landing page on the very first night using a Bootstrap template backed by a Lua script hosted on Webscript.io.  The Webscript.io platform served as a great, quick, database-free way to get email addresses of interested customers and to send them a fast "Welcome" message.

We didn't come out of the weekend with any awards, but we did close out Sunday with a ton of knowledge.  My biggest takeaway came in the form of team dynamic; while developers at Startup Weekend events are typically harder to acquire, our team containing five almost ran into the opposite issue of having too many developers.  A single developer could have played the role of product manager and more effectively divvied up the workload for the other programmers.  This would have been a huge help in preventing overlap and improve efficiency while building.

The Roanoke Times has a great writeup on the weekend, which includes more information about Pet My Pets.  If you're interested in the idea, you can check out the Pet My Pets website at http://petmypets.co.

A huge thanks goes out to Alex Meng, Viktoriya Leshchenko, David McGrath, and Lauren Rose for offering up their pets for the production of the video at the top of this post.

[amussey]

Rackspace Monitoring Agent Install Script

Edit: As Shane pointed out in the comments, there is a new Install Agent feature in Cloud Intelligence that offers a similar one line agent install.  Visit Cloud Intelligence to check it out!

After recently setting up a new Rackspace Cloud Server with the Cloud Monitoring agent, I was annoyed by the length of the install Knowledge Center article.  I decided to write a script to assist in and speed up the install of this agent.

Before running this script, you will need the following:

Your username.

Your API key (found on the MyCloud "Account Settings" page).

curl installed on your server.

To install the Cloud Monitoring agent, simply run this command from your Cloud server:

sudo bash -c "$(curl -s https://raw.githubusercontent.com/amussey/blog-posts/master/2014/10_Rackspace-Monitoring-Agent-Installer/install.sh)"

This script has been tested with all currently available Performance flavor Cloud servers that have OSes supported by the Cloud Monitoring Agent Knowledge Center article (as of Oct 2014).  For more information on compatibility and to view the contents of the script, visit the github repo for this blog post.  As always, pull requests and comments are welcome!

[amussey]

I recently open sourced the code for the website I use for DJ booking, DJMu-Z.com.  On the front page, I host this mouse responsive "M" animation using a single image and a piece of Javascript.  You can check out the code involved on Bitbucket:

https://bitbucket.org/amussey/dj-mu-z-website

[amussey]

During the summer, Rackspace Blacksburg set on a consistent schedule of office-wide tech talks.  These talks ranged in subject from conference presentations to new technologies to managerial styles.  Above are some of the talks that have been given in the last couple months that I have had a chance to record and post to the Racksburg YouTube channel.

You can watch all the Racksburg Tech Talks via the Rackspace Tech Talks playlist.  You can see more content from the Rackspace Blacksburg office on the Racksburg YouTube channel.

[amussey]

The Rackspace Blacksburg (Racksburg) office.  Taken for the Racker Rush website.

[amussey]

Lightweight Lua JSON Validator

I'll admit it: I've been completely sucked into webscript.io.  Webscript.io is a site that allows for small Lua scripts to be hosted on user-selected URLs.  A coworker and I have spent the last two weeks designing new dashboards for our Rackspace team, scraping websites, and throwing small scripts back and forth.  The number of written lines of Lua between the two of us has easily broken into the thousands.

While working on a recent script, I ran into a situation where JSON would need to be input and parsed.  Unfortunately, the built-in webscript.io JSON.parse function does not have any sort of decent error handling.  If there is a problem, your script 500s and the user receives a blank response.

There are a lot of incredibly in-depth Lua JSON libraries, but keeping with webscript.io's spirit of "small scripts", I was hesitant to import any of them in their entirety.  All I needed from these libraries was validation - is this string valid JSON, or are there missing brackets and broken syntax?

This led to the creation of a Lua JSON Validator.  The module does exactly as described above: takes a string as input and returns true or false if it's valid JSON.

To use the module, you simply need to import it via a require statement:

local validJson = require "amussey/lua-json-validator/validJson" if validJson(jsonString) then -- interact with valid JSON else -- report invalid JSON end

A series of test cases are also provided with the module.

To view the source code, visit the github repo:

https://github.com/amussey/lua-json-validator/

As always, pull requests on the repo or comments on this post are welcome!

[amussey]

I had the opportunity to film the Racksburg (Rackspace Blacksburg) response to the ALS Ice Bucket challenge from Graham Weston last week.  Awesome to see so many participating Rackers!

[amussey]

Converting HTML Colors from Hex to RGB/RGBA

While constructing the Github Team Pull Request Highlighter, I ran into the need to translate color codes back and forth between hex (#ff0000) and RGB/RGBA (rgba(255, 0, 0, 1)).

This led to the construction of a micro-library hex-to-rgb.js (direct link to the library).  This library allows for easy conversion between the formats through two functions: hexToRgb and rgbToHex.

hexToRgb returns an object with r, g, and b properties.

var color = hexToRgb("#1fbcff"); color.toString(); // rgb(31, 188, 255) color.r; // 31 color.g; // 188 color.b; // 255

hexToRgb can also accept an alpha value has a second parameter. This allows the output of rgba strings.

var color = hexToRgb("#1fbcff", 0.5); color.toString(); // rgba(31, 188, 255, 0.5) color.r; // 31 color.g; // 188 color.b; // 255 color.alpha; // 0.5

rgbToHex returns a hex string in the format #ffffff. It can accept a wide variety of inputs.

rgbToHex(31, 188, 255); // #1fbcff rgbToHex("rgb(31, 188, 255)"); // #1fbcff rgbToHex("rgba(31, 188, 255, 0.5)"); // #1fbcff

rgbToHex can also accept the color objects produced by hexToRgb.

var color = hexToRgb("#1fbcff"); rgbToHex(color); // #1fbcff var color2 = hexToRgb("#1fbcff", 0.5); rgbToHex(color2); // #1fbcff

The repo for this library is available here (or click here if you want save time and go directly to the library).

[amussey]

New UI & Better Performance in Get Drunk Not Fat!

A little over a year ago, I posted about a secret project I had been working on: the official iOS and Android app for Get Drunk Not Fat.  There have a been a couple big updates since then, but today I have another to show off.

The latest interface update inherits the flat-UI style of iOS 7, along with a more appealing color scheme.  Along side that, the performance has been improved, so you should see all around speed ups on your device.

The latest version of Get Drunk Not Fat is available right now through the iTunes Store and the Android Market.

[amussey]

Let's Code Blacksburg! has been doing some truly awesome work introducing technology to the youth in the NRV. I had the chance to film at two of their recent events (an intro to PyGames class and a beginners Arduino class), and really saw a ton of cool projects result.

[amussey]

Improving Page Performance on Sites with Many Embedded YouTube Videos

While working with videos for a couple blogs, I've run across performance issue while embedding multiple videos on a page.  Once you get past 2-3 embedded videos on a page (something that isn't terribly challenging if that's a primary focus of the site), the scroll speed slows to a crawl and over-all user performance begins to deteriorate.

To mitigate these problems, I spent some time over the weekend while at the Rackspace Blacksburg Hackathon creating a small javascript library, youtube-speed.js, that will replace embedded YouTube iframes with static images until the user wants to watch the video.  When a user clicks on the image, the video loads from YouTube and begin playing.

Here's how the page appears with and without the library:

Before

After

This requires no modification to the existing code (unless you're adding custom styling to the iframes).  All you need is to embed the script:

<script src="youtube-speed.js"></script>

The source for the library is available with demos on github.  If you're in a rush, here's a direct link to the library.

If you run into any bugs or problems, let me know in the comments or on the github page!

[amussey]

HTML5 Custom Markdown Tag

I was inspired by a Tech Talk given by Brad Gignac a couple months ago at Rackspace on custom Web Components, so while looking at a way to cleanly parse out Markdown inside of an HTML document, I turned to custom HTML5 tags.  To my dismay, I couldn't find anybody implementing something like this for markdown.

To fix this issue, I've put together a small library to implement a <markdown> tag.  Since these custom elements require a dash to be included, I've had to use <code-markdown> instead.

The library is available for download on github.  If you would like to see an implementation example, read on after the break.

Implementation is simple: just include the Javascript library in the head of your file (Showdown.js is currently being used for parsing, but is no longer required to be explicitly included):

<script src="markdown.elem.js"></script>

Then, include your Markdown tag:

<code-markdown> # Markdown test We can make stuff **bold**, *italicized*, even put in [links](http://amussey.com)! ``` def winning(self): try: self.name = 'Andrew' except Exception as e: print e ``` <code-markdown>

This will render out to look as such:

The Markdown output within these tags will conform to John Gruber's original Markdown spec, allowing it to be easily stylized with the rest of the page.

The Javascript library to run this is available on github.  As always, comments and pull requests are welcome!

[amussey]

Chome Extensions: Modifying the DOM from an Extension popup (Message Passing)

I've spent a lot of time in the last couple months learning my way around building Chrome extensions, resulting in a couple useful extensions for day-to-day work.  One of the most vital pieces of knowledge was understanding how to modify the DOM when something in the popup of an extension was clicked or modified.  This involves a series of somewhat confusing message passes from the popup to the background script to a content script on the page.  The following explains that process.

We'll start by setting up a listener in the content script that is going to be included on the page the extension is working with.

content_script.js

chrome.runtime.onMessage.addListener( function(request, sender, sendResponse) { alert(request.message); } );

In this case, the listener is just going to throw an alert popup with whatever message it receives.  This file will have access to the DOM on the page, so you can do anything from changing the color of fonts to replacing all of the images with cats.

The message this content script is listening for will be coming from a background script.  We can define this call as a function:

background.js

pop_alert = function(passed_message) { chrome.tabs.query({active: true, currentWindow: true}, function(tabs) { chrome.tabs.sendMessage(tabs[0].id, {message: passed_message}, function(response) { /* Ignore any response. */ } ); }); }

Finally, we'll tie this all together by putting the code to call this function in the popup:

popup.js

var bg = chrome.extension.getBackgroundPage(); bg.pop_alert("Passing a value");

This could be fired from anywhere: on a button down, clicking a link, or during the firing of an event.

That's all there is to it!  I've provided my manifest.json file below to help incase you have any issues while connecting these three scripts.

manifest.json

{ "name": "Long Extension Title", "short_name": "Short Name", "version": "1.0.0", "manifest_version": 2, "description": "Description of the Extension", "icons": { "128": "icon.png", }, "browser_action": { "default_icon": "icon.png", "default_title": "", "default_popup": "popup.html" }, "background": { "scripts": ["background.js"] }, "content_scripts": [{ "matches": ["https://amussey.com/*"], "js": ["content_script.js"], "run_at": "document_end" }], "permissions": [ "tabs", "https://amussey.com/" ], "web_accessible_resources": ["popup.js", "content_script.js"] }

[amussey]

Docker: Starting Docker on System Boot on OSX via boot2docker or docker-osx

While it's a relatively minor annoyance, i wanted to find away around the requirement of launching boot2docker, the LXC-emulating layer for OSX, every time I rebooted my machine.  The following scripts have been designed to automate this process and eliminate the need to manually start that machine.

To set this system up, run the following command:

curl -sS https://raw.githubusercontent.com/amussey/blog-posts/master/2014/05_OSX-boot2docker-on-Startup/setup.sh | /bin/bash

For a detailed description of what this script is doing, you can check out the write up after the break!  (I'd recommend at least glancing over it so you're not blindly running shell scripts from the internet.)

How the Script Works

We'll start by pulling in docker-osx.  This is an excellent all-in-one script that helps automate a lot of the setup and launching process for the Virtualbox VM.

curl https://raw.githubusercontent.com/noplay/docker-osx/0.10.0/docker-osx > /usr/local/bin/docker-osx chmod +x /usr/local/bin/docker-osx docker-osx start

By starting this scirpt, we'll begin the process of downloading a base boot2docker image and setting up Vagrant.

Next, we'll pull in the script that will be run when you boot your Mac.  This script will be stored in /Applications/Docker.

mkdir /Applications/Docker/ curl https://raw.githubusercontent.com/amussey/blogs/initialize_docker.sh > /Applications/Docker/initialize_docker.sh

To launch that script, we'll need a .plist file that will be incldued in the boot sequence.   The contents of that file can be viewed here.

curl https://raw.githubusercontent.com/amussey/blogs/com.user.boot2docker.plist > ~/Library/LaunchAgents/com.user.boot2docker.plist launchctl load ~/Library/LaunchAgents/com.user.boot2docker.plist

Finally, we'll make sure the docker environment information is included in the bash profile.

if [ $(cat ~/.bash_profile | grep "eval \`docker-osx env\`" | wc -l) -eq 1 ] ; then echo "eval `docker-osx env`" >> ~/.bash_profile fi source ~/.bash_profile touch ~/.docker-osx-boot-launcher.log

That's all there is to it!  As always, feel free to leave comments or make modifications with a pull request.

[amussey]