Developmentables

A blog by the development team of Inventables var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); try { var pageTracker = _gat._getTracker("UA-1939094-4"); pageTracker._setDomainName(".developmentables.com"); pageTracker._trackPageview(); } catch(err) {}

[developmentables-blog]

AJAX Facebook Like Button

We recently added Facebook "Like" buttons to each of the product pages, but it significantly slowed down the page so we decided to implement the AJAX version of the Facebook Like Button.

Here are the steps to setup the Like button to load asynchronously:

1. Add the Facebook HTML element to the page:

2. Add Javascript to insert and load the Facebook Developer SDK asynchronously:

3. Add Facebook Open Graph meta tags to your pages so your Facebook Fans will receive clean product names and images for the pages they like. This is an example of what we added for each of the products:

4. Test and validate your Open Graph tags via the Facebook Linter 

5. Take a look at your page.  It will load just like normal:

  Then magically, the Facebook Like button will appear moments later:

[developmentables-blog]

Relating Inventables Products with Hadoop MapReduce

If you're a frequent user of e-commerce websites, you're probably familiar with the lists of related products these sites often feature to help customers find what they're looking for on the store. [Amazon](http://amazon.com), in particular, sports several such lists on every page, including "Frequently Bought Together" and "Customers Who Bought This Item Also Bought." If you've been to our site lately, or read [our announcement](http://inventables.blogspot.com/2010/09/new-beginning.html), you know that [Inventables](http://www.inventables.com) is now an e-commerce site. As such, we thought it would be a good time to explore adding some of these types of lists to our product pages. In this post I'll talk about how we scripted a process that uses [Hadoop MapReduce](http://hadoop.apache.org/mapreduce/) to parse the millions of product views we've accumulated over the past year or so, and put it into a format that can be used to display the list "People who viewed this product also viewed" for every product on our site. To see examples of this list in the wild, check out one of [our](http://www.inventables.com/technologies/squishy-gel-magnet#viewed-technologies) [cool](http://www.inventables.com/technologies/bend-sensor#viewed-technologies) [products](http://www.inventables.com/technologies/rubber-glass#viewed-technologies). ### Formal Definition of our List Given two products `p1` and `p2`, we will define `p1 related-by-path-to p2` (and vice versa) if there is exists a user-session-specific path P where `page-view(p1)` and `page-view(p2)` are both members of P. Additionally, the weight of this relationship between `p1` and `p2` will be equal to the number of distinct paths for which they are both members. If 10 users viewed both `p1` and `p2` during a single visit, then those two products will have a path-relation of weight 10. To display the list for a given product, we will sort the set of related products in descending order of weight. For example, let's say you're looking at [Rubber Glass](http://www.inventables.com/technologies/rubber-glass). If you see [Flexible Metal Brick](http://www.inventables.com/technologies/flexible-metal-brick) as the first item in the People who viewed Rubber Glass also viewed list, then that means more people who looked at Rubber Glass looked at Flexible Metal Brick than any other product on the site. ### The Raw Data To build our list we'll be drawing on a single table from our application database. This table stores an entry for every unique product viewed by a user during a given session. Only two columns from this table are needed for our task: `path_id` and `product_id`. We can export this data from our [Amazon RDS](http://aws.amazon.com/rds/) MySQL instance, using the mysql command line utility like so: mysql -u $user --password=$pass --host=$host --batch -e \ "SELECT path_id, product_id FROM recent_items" \ $database > /tmp/input/recent_items.tsv This tab-separated file will constitute the initial input to our MapReduce workflow. ### Tool Selection I selected [Hadoop Streaming](http://hadoop.apache.org/mapreduce/docs/current/streaming.html) to drive the MapReduce workflow because of the large volume of data we're dealing with, and also because we run our application infrastructure on the [Amazon AWS](http://aws.amazon.com/) platform, which, in the form of their [Amazon Elastic MapReduce](http://aws.amazon.com/elasticmapreduce/) offering, has built-in support for Hadoop Streaming. For now we can get by with running Hadoop in [single node mode](http://hadoop.apache.org/common/docs/current/single_node_setup.html), but if our scale grows or we start doing more things with MapReduce, we'll be able to easily switch to true [cluster mode](http://hadoop.apache.org/common/docs/current/cluster_setup.html). ### Solution Outline The goal of this exercise was to start with a table of `path_id, product_id` values and ultimately generate a new table of `source_product_id, target_product_id, weight` values, where the source and target product are related to each other with the given weight. From this final data format, it is trivial to build a gallery to display the top n related products for a given source product. To get from the millions of `path_id, product_id` records to the desired `source_product_id, target_product_id, weight` format using MapReduce, we take a two-phase approach. ### MapReduce Phase 1 This phase creates an output file where each row represents one instance of a product-to-product relationship from a single path. As such, each row will carry a weight of 1. ## Map: The input to this function is already in the desired format of `path_id, product_id`, so this mapping step is just the identity function. ## Reduce: Hadoop MapReduce guarantees that all rows for a given key emitted by the map step will go to the same reducer. In our case, the key is a path id. So in this first reducer, we iterate over all the rows for a given path, appending each product id to an array. Once we have all the product id's for a given path, we can emit all of the permutations of `product_id, product_id` pairs. For example, let's say the input included the following rows: path1, product1 path1, product2 path1, product3 (The user looked at 3 products during his session.) The reducer would first map `path1` to the array `[product1, product2, product3]`. Then, it would emit six key/value pairs of the form `source_product_id_target_product_id, weight`: product1_product2, 1 product1_product3, 1 product2_product1, 1 product2_product3, 1 product3_product1, 1 product3_product2: 1 And the ruby source: ### MapReduce Phase 2 The second and final MapReduce will sum the weights associated with every unique `product1_product2` key. ## Map: Since the input to the second map phase is already in the desired format of `product1_product2, weight`, the identity function can also be used here. ## Reduce: The reduce step, being given all the weights for a given product-product combination at a time, simply iterates over each key, sums the weight, and emits one key/value pair with the total. Ruby source: ### Pulling in the data After the second MapReduce phase completes, we have the data we need. All that's needed is to import it into our database. We can accomplish this again with the mysql command line utility: mysqlimport --local --compress -u $user --host=$host \ --columns=source_product_id, target_product_id,count \ --replace $database \ /tmp/final_output/viewed_products.tsv Note: the `viewed_products` table has a unique index on `source_product_id, target_product_id`. ### Stitching it all up We want our process to be automated so that it can be run on a schedule. This can be accomplished with a [simple shell script](https://gist.github.com/720445#file_gistfile3.sh). ### Conclusion As you can see, it wasn't hard at all to build our first list of related products using Hadoop MapReduce. We plan to do a lot more in this realm going forward. --Jeff

[developmentables-blog]

How Startups are like Soccer Teams

 This summer I watched a lot of [World Cup soccer](http://www.fifa.com/worldcup/index.html) games. Initially, I only watched the U.S. soccer team's games, but then I slowly expanded my viewing. I started watching all of Spain's games because I have close friends in Spain. Then I started catching the replays of as many games as I could in the evenings, and I'd sometimes listen to some of the audio of games during work if it was a particularly compelling matchup. Prior to this World Cup, despite playing in recreational soccer leagues for nine years as a kid, I was pretty ignorant of the game inside the game, especially at the top levels of competition. While watching the games this summer, though, I really started to soak up the details. I learned about different strategies like pushing your players up and attacking vs. playing a defensive game while looking for chances to counter-attack. I learned about tactics like trapping players [offsides](http://en.wikipedia.org/wiki/Offside_(association_football)) or taking some [corners short](http://en.wikipedia.org/wiki/Corner_kick#Short_corner) to open up the middle of the field. But the biggest thing that struck me in watching these games was how well the players of different positions worked together within a team. When I was a kid playing in those recreational leagues, players at different positions hardly worked together at all. Forwards would not help defenders. Defenders would not help forwards. Most of the time a team was little more than a group of individuals all playing their own game.

That's not how it works at the world cup level. Sure, the players fill the same positions we did based on what they do best: strikers are best at moving the ball past defenders and scoring goals; midfielders are best at maintaining possession of the ball and setting up their teammates for goal scoring chances; defenders and goalies are best at neutralizing an attack and recapturing possession of the ball. But what sets the players apart at the top levels of soccer, other than sheer talent, is that they spend a lot of time doing things that are _not_ the things most often associated with their position. Forwards spend a lot of time helping neutralize an attack, and even defend shots on goal when there is a penalty kick or a corner kick attempt. Midfielders may attack and score goals, or they may fall back and help their defense deal with a threat. Defenders may push the ball up the sidelines on an attack, or come all the way into [the box](http://en.wikipedia.org/wiki/Penalty_area_(football)) to field a corner and put a shot on goal. Each player's position is really just a _default_. It's what they spend most of their time doing and are primarily responsible for, but in order to make the team effective at an elite level, each player must spend a lot of time outside their core area of expertise. There are two big corollaries to this model in the startup world. First, there is a high correlation in the assignment of core responsibilities. On the front lines, you have the individuals who are responsible for customer development. They spend most of their time engaging the enemy (the customer)*. If your startup is a soccer team, they are the forwards. On the other side, you have your development team. Their responsibility is to neutralize problems by delivering features that will give the customer development team a chance to pivot and try a fresh angle. In the soccer model, they are the defense. In the space between these two units roam the business analysts. They help the development team convert problems into solvable units and then make sure the completed features lead to data and insights that funnel smoothly back to the customer development team. They are the midfielders. The second big corollary is that just as with soccer teams, the startup that enforces rigid divisions of labor is doomed to fail. To be truly elite, a startup must emulate the fluid boundaries between roles demonstrated by the elite soccer team. The development team needs to understand the customer's problems and assist with designing and administering usability tests and developing personas. A developer may even need to propose a business model hypothesis (take a shot on goal). The customer development team needs to help the development team, if not by actually getting into the code and building a feature, then at least by listening to developers and working with them to simplify and distill features down to a size that can be executed consistently and efficiently. Business analysts need to take an active role in customer development by assembling and distributing surveys and questionnaires, conducting formal customer interviews, as well as crafting and administering usability tests. They might have to go further to retrieve insights (the ball) than they would like by building spreadsheets to evaluate usage patterns or crafting and administering A/B tests of a given feature. What this exploration of the similarities between soccer teams and startups highlights, of course, is the importance of teamwork--a specific kind of highly collaborative and help-based teamwork. If you're involved in a startup, I hope this discussion serves as a fresh source of inspiration. And I hope your team takes home the next World Cup.

—Written by Jeff

—Illustrations and Title Photo by Billy

If you'd like, you can [fork our jerseys on github](http://github.com/wwhited/Startup-Jerseys). \* If you are familiar with Jeffrey Moore's "[Crossing the Chasm](http://www.amazon.com/Crossing-Chasm-Geoffrey-Moore/dp/0060517123)”--where penetrating a market is likened to launching a D-Day invasion--you will no doubt be used to thinking of customers as "the enemy"

[developmentables-blog]

On Batteries and Brainstorms

_"I'm fully charged by this idea."_  I re-read Mark Boulton's [A Practical Guide to Designing for the Web](http://fivesimplesteps.com/books/practical-guide-designing-for-the-web) this week and was inspired by a bit of wisdom from his 'Ideas' chapter. In this chapter, Mr. Boulton lays out a very useful strategy for conducting brainstorm meetings that involves ranking generated ideas using what he calls a 'Passion-O-Meter.' By his own admission, this is just a "fancy name for using some stickers", but I was struck by the potential of the idea to offset emotions that often accompany such an exercise. _Have you ever been in meeting where someone got offended because their idea was ill-received? I have._ Because Mr. Boulton did not provide any explicit instructions (or ready-made stickers for that matter), I was required to develop my own version of the 'Passion-O-Meter.' I wanted to make the ranking process as fluid, fun, and free of conflict as possible. So the first thing I had to decide was what to actually put on the stickers themselves. In my experience ideas generally fall into 1 of 3 categories: + Great + Good + Not So Good Since I knew the stickers were going to be physically affixed to the ideas themselves (we write them out on a giant 20ft X 20ft whiteboard) I had to make sure they weren't egregious in any way. No 'loser stamps' allowed. At the same time, I wanted something clever and memorable. For some reason, my mind very quickly jumped to the idea of battery meters. Maybe I was inspired by the errant behavior of my iPhone 3G's battery since updating to the latest OS. _Who knew how much the fluctuating charge level of an iPhone battery could affect one's mood?_ Inspired by the iPhone's battery meter icon and [Simple Bits' Charge T-Shirt](http://shop.simplebits.com/product/charge-tee), I created icons and phrases for three possible rankings:  After creating the artwork, the next step was to import the images into an Avery template, in this case #5160. I dubbed my version of the idea ranking process the 'Passion-O-Meter _Blitz_' as the team was only given 5 minutes to rank about 30 ideas. I wanted there to be a flurry of activity so that each team member's votes could be more or less anonymous. Based on the feedback I received after the meeting (and the efficacy of the ranking procedure itself) it would seem that the battery stickers and my interpretation of Mark Boulton's 'Passion-O-Meter' exercise were a success! If you'd like to use the battery stickers I created in your own brainstorm meetings, please feel free to download the Adobe Illustrator source file from my repository on GitHub: [ http://github.com/wwhited/brainstorm_battery_stickers/tree/](http://github.com/wwhited/brainstorm_battery_stickers/tree/) —Written by Billy

[developmentables-blog]

Diffing Darwin: Visualizing the Future of a Web App

__Could there be more reciprocity between the way programmers and designers think than what most in our industry have traditionally believed?__  In a planning meeting this week, our development team discovered a surprising similarity in the way our designer, Billy, and our lead developer, Jeff, visualize things. As our CEO discussed his vision for the company over the next several months, Jeff and Billy were both struggling to get a vivid picture of each of the 3, 6, and 9 month sign posts he was laying out. So we both spoke up about how those sign posts could be made more clear. First, Billy said he pictured the website as a series of time-lapsed screenshots similar to the way human evolution is often portrayed on the Darwinian scale. He then said that to explain the signposts he would simply pluck an image from the series created in his head and describe it. While Jeff liked this approach, he said that his picture would center more around the _differences_ between each sign post. He visualized a set of high-level __git diffs__ showing the features that had been added and subtracted between signposts. At first, the _Programmer vs. Designer_ contrast seemed so cliche and obvious that it was comical. But after a few quick moments of reflection, the core similarities between the two approaches dawned on everyone. It became apparent that if you were familiar enough with the features of a website, a git diff could serve as a blueprint from which you could construct a visual image. We decided to illustrate our revelation by 'git diffing' the 'visual scale of human evolution'. At Inventables, our designer sits right next to, and works directly with, our programmer. Here's to hoping that more companies abandon the "over the wall" approach to designer/developer collaboration:        —Written by Jeff and Billy PS - If you'd like to build a better monkey, the [source code](http://github.com/inventables/evolution) is on github.

[developmentables-blog]

What Motivates us at Inventables

Lately there have been a lot of headlines about what motivates us in the workplace. In [this talk](http://www.youtube.com/watch?v=u6XAPnuFjJc&feature=player_embedded), Dan Pink boils it down to two things: 1. You make enough money so that you're not worried about money 2. You have the autonomy to direct your own work and make an impact Dan then gives an example of an amazing place to work where one day a quarter the employees get to work on whatever they want. The only condition is that they have to present their work at an informal, fun type of gathering. This company, the Australian based [Atlassian](http://www.atlassian.com/), has reaped a lot of benefits from this approach, including bug fixes and new features that they wouldn't have otherwise had the time to build. Based on this data, I'd say Atlassian does sound like a good place to work. At Inventables, we don't have one day a quarter to work on whatever we want. We have _one day every week_. Instead of 4 days a year, we have 50 days a year. __50 days to work on whatever you want__. And instead of having to present your work at a meeting, you don't have to present your work at all. Unless you want to. And that presentation can take any form you want. Maybe you want to have an informal, fun type of gathering. Maybe you just want to send out a link. Because we build software using an agile approach, we've dubbed our one day a week of free exploration "Free point Friday." And we, too, have reaped a lot of benefits from our approach. Like Atlassian, we've fixed some bugs. We've built some features. We've redesigned the look and feel of our website. We've built entirely [new applications](http://oneth1ng.net) just for fun. We've mined data looking for patterns, caught up on reading, and just played around with [new](http://en.wikipedia.org/wiki/HTML5) [technologies](http://nodejs.org/). As the leader of our technical team, I believe our "Free point Friday" approach is the best way to run our business because if someone has an idea for how to improve things, they don't have to sell it. They don't have to set it aside for later. They can just do it, and they're free to fail. The failure will not be analyzed or filed away and brought up later. Our business is about discovery and innovation. We believe anything's possible, and we believe that if you just let people explore, they'll prove it to you. —Written by Jeff

[developmentables-blog]

Goodnight, IErene.

As of May 2010, Inventables will no longer support Microsoft's Internet Explorer 6. The reasons for this are numerous and varied and range from the technical, to the methodological, to the ethical, to the practical. IE6 is a 9 year old piece of software. That's at least a century in internet years. At a time when even resource-laden internet powerhouses like Google and Amazon are discontinuing support for the aged, buggy, and categorically unsafe browser, we simply felt it was time to say goodnight. Still, we acknowledge that there are many users (namely at large corporations with staggering IT infrastructures) that have no choice but to use IE6. We understand the predicament these users find themselves in, but given the small size of our development team and the ever changing nature of our app, we found we could no longer afford to spend valuable development time getting IE6 to comply. Using Dean Edwards IE9.js, we've managed to make IE6 "hang together" for now, with minimal effort. This has made developing for IE in general much easier, and has provided those using IE6 a somewhat usable consolation. Here's a screenshot of what IE6 users can expect:  Let's let the web evolve...it's ready and willing! In this spirit I am reminded of Bob Dylan's immortal lyrics: Your old road is / Rapidly agin' / Please get out of the new one / If you can't lend your hand / For the times they are a-changin'. —Written by Billy

[developmentables-blog]

Adventures in Typekit

One of the many great things about working at Inventables is that we, the development team, are continually afforded the opportunity to experiment with—and even implement—cutting edge web technologies like CSS3, HTML5, Typekit, and many others. Because we're small and nimble we are able to adapt quickly to the changing landscape of the modern web. It's a great atmosphere in which to create. But while being at the cutting edge is great for our team's morale, we must balance our excitement with restraint. Using advanced and not-always-fully-supported web technologies is an exercise in trial and error, frustration and reward. Case in point: [Typekit](http://www.typekit.com), the web-font hosting service that has been gaining popularity in recent months. **Where Typekit succeeds** Typekit works well as a repository for high quality, professional and legal web-ready (though not always "web-optimized") fonts. Because Typekit has struck partnerships with several well established foundries they are able to serve up many excellent, complete and otherwise unavailable (due to strict licensing agreements) fonts that free-font sites like [fontsquirrel](http://www.fontsquirrel.com) simply cannot. As a result, Typekit takes the guesswork out of choosing custom web fonts. I can browse the site and quickly/easily identify a host of viable fonts without having to worry about EULAs and without being restricted by a limited number of available font weights. Until the [WOFF](http://en.wikipedia.org/wiki/Web_typography) takes off, Typekit and similar font-serving sites like [Font Deck](http://fontdeck.com/) provide the best options to those designers looking to use fully-featured, custom fonts on their websites. **The Unfortunate FOUT!** The biggest frustration I currently have with Typekit is with what is known as the "flash of unstyled text," or FOUT for short, which is an issue that effects websites that use font-linking. The FOUT is an unsightly effect that occurs because of the fact that linked fonts are loaded, like images, after all the other content on a webpage has been loaded. As a result, your back-up fonts (as specified by your CSS font stack) will flash for a few milliseconds before your typekit fonts load. Due to the spatial difference between various typefaces, this produces a distracting "visual hiccup" that can make a site look broken even to the most oblivious of non-designers. To put it bluntly, the FOUT sucks. It is for many who would otherwise commit to a Typekit service plan, an outright deal breaker. Now unfortunately for the folks at Typekit (who are painfully aware of this problem and taking on much of the blame), the FOUT is an issue that affects everyone using linked fonts whether they employ a third-party solution or host fonts themselves. As far as I've been able to tell, it's a browser based issue. Typekit performance appears to vary, to some degree anyway, between Firefox, Chrome, Safari, and IE. Nonetheless, much to my disappointment, the only way I found to effectively manage the FOUT was to significantly reduce the file size of my "type kit" to around 75k, or just one font with two weights. So much for my dreams of versatility! With that said, Typekit is working on improving this issue and they have made some experimental solutions available that seem quite interesting. However, considering the monthly cost of the service, I am really hoping these solutions move beyond the experimental stage in the very near future. To keep tabs on Typekit's response to the FOUT, check out these threads: [http://getsatisfaction.com/typekit/topics/avoiding_the_fout](http://getsatisfaction.com/typekit/topics/avoiding_the_fout) [http://getsatisfaction.com/typekit/topics/flash_of_unstyled_content](http://getsatisfaction.com/typekit/topics/flash_of_unstyled_content) **So, The Empire Strikes Back...you're gonna proof those fonts right?** Another equally serious issue I've had with Typekit surrounds the display of certain custom fonts on Windows (mainly while running XP). Windows uses a system called Cleartype for anti-aliasing fonts which needs to be turned on in order for font smoothing to occur. In Windows 7 and Vista, Cleartype is turned on by default and Typekit fonts look more or less like they should (if we use OS X as a benchmark). But in Windows XP Cleartype is turned off by default which can mutate the appearance of certain fonts and make them look jagged and unreadable. Again, this is an issue that the folks at Typekit are not directly responsible for as the problem is rooted elsewhere (in this case, both in Windows XP's native font rendering settings and the actual "hinting" of the offensive fonts). Nonetheless, it is a real problem that can affect a design in truly serious ways. And though Typekit does offer a robust "type tester" feature that does a pretty good job at approximating what a chosen typeface will look like in each browser, I believe Typekit could do more to advise it users on which fonts are best suited for use in Windows and which are not. Perhaps by flagging troublesome or poorly hinted fonts? For a discussion of this problem, check out these threads: [http://getsatisfaction.com/typekit/topics/typekit_fonts_rendering_horribly_on_windows_based_systems](http://getsatisfaction.com/typekit/topics/typekit_fonts_rendering_horribly_on_windows_based_systems) [http://getsatisfaction.com/typekit/topics/only_readable_if_cleartype_is_enabled](http://getsatisfaction.com/typekit/topics/only_readable_if_cleartype_is_enabled) **Conclusion** For the time being, we are still using Typekit here at Inventables, despite its shortcomings. We believe in the "case for real fonts on the web" and prefer the greater freedom and flexibility that using custom fonts provides. However, despite our hopes for Typekit, we believe it still has some issues to work though before it becomes more widely adopted. —Written by Billy

[developmentables-blog]

Amazon RDS Redux: Tracking down a slow query

In our [last post](http://developmentables.com/post/499001044/migrating-inventables-to-amazon-rds) we talked about how Inventables migrated from a MySQL server running locally on an EC2 instance to the [Amazon RDS service](http://aws.amazon.com/rds/). One benefit of using Amazon RDS is that you gain access to a large array of performance statistics provided by the [Amazon CloudWatch](http://aws.amazon.com/cloudwatch/) service without having to do any extra work. We were eager to start viewing these statistics to see how the new platform was doing. ## Checking out our stats The CloudWatch command line tools are a breeze to [download](http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2534&categoryID=88) and install locally--you don't have to run them on your EC2 instances. In just a few minutes we were able to view the minimum, maximum and average CPU utilization, available storage space, and number of database connections for any period over any time frame since the launch of our RDS database instance. By running the following command, we immediately we noticed that our CPU utilization was maxing out in the mid-to-high 90's every hour: The numbers seemed too high. Something was amiss. ## Tracking down the cause We knew we had several options for investigating the problem, but since we were new to RDS we decided to use this as an opportunity to improve our familiarity with some of the core features of the system. We developed the following plan of attack: 1. Launch a clone instance of our production RDS database 2. Turn on slow query tracking for the clone database 3. Launch a new, standalone EC2 instance running the same code as our website, but pointed at our clone database 4. Exercise the standalone instance and then check the MySQL [slow query logs](http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html) to see if we had any bad queries This is why we love AWS: with a few simple steps we can produce a brand new environment that looks and acts just like our production environment at all levels, with the same data. If a query is only taking a long time when there is a lot of data, we'll find it. And we won't be tampering with our real site in the process--it will happily hum away, utterly oblivious to our performance experiments. ## Attack of the clones Launching the clone database instance was effortless. As with our initial migration to RDS, we decided to use [the ruby client](http://github.com/grempe/amazon-ec2): Turning on slow query tracking took a little more effort. This required [creating a new DB parameter group](http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2935) for our clone database instance. Our production instance uses the default parameter group, so we didn't need to copy over any custom settings. We just set up our new group to flag any queries executing for more than 1 second on any table with more than 100 rows: With slow query tracking turned on, we launched and configured our EC2 instance and manually walked through a few user workflows to exercise the most commonly used parts of our site. Then we checked the slow query log on our clone database instance: select * from mysql.slow_log; Bingo. The result set included one entry. We fired up a mysql command line client and ran EXPLAIN on the bad query, which showed that we were doing a full table scan on a table with several hundred thousand rows. ## Follow-up After adding the appropriate index, we kept a close eye on our maximum hourly CPU utilization in our production environment. It immediately dropped to the low 80's after the fix and has been holding steady at or below that level since. Catching index problems in production is definitely not our first choice. We're reviewing our approach for identifying needed indexes at development time and how we can improve on it. If you have any suggestions, let us know! —Written by Jeff

[developmentables-blog]

Migrating Inventables to Amazon RDS

Background

The Inventables Marketplace and all the technology needed to support it reside and operate in the cloud. Specifically, we use the Amazon Web Services (AWS) offering, and we're really happy with it.

Our site is a Ruby on Rails application that runs on two Amazon Elastic Compute Cloud (EC2) instances. All of our static assets, such as images and videos, are stored on the Amazon Simple Storage Service (S3).

Until this past weekend, one of the EC2 instances, in addition to being an application server, also formed our database tier. It ran a local MySQL installation. All data was stored on a locally mounted Amazon EBS volume. A cron job would run twice daily to take consistent snapshots of the volume and store them on S3.

This architecture performed beautifully. It had no trouble handling our current load volumes and never went down. But it did have a few limitations:

Hard to grow

Because of its nature as an EBS volume, our database was constrained to a static size. Allocating too large a volume is wasteful and expensive. Expanding to a larger volume would require a number of steps: bringing the site down; snapshotting the database; creating a new, larger volume from the snapshot; and finally re-attaching and re-mounting the new volume to the instance. Because of the tools offered by AWS, including the AWS Management Console, each of these steps would be relatively easy, but they are still required and there are a number of them. Yes, you could probably script them, but there is still a decent amount of heavy lifting required to expand to a larger EBS volume.

Hard to Scale

First, because it ran locally, our MySQL instance was constrained to the compute capacity of the EC2 instance hosting it. Expanding its capacity would require terminating that instance, starting up a new one, and reconfiguring our Elastic IPs.

Second, the MySQL instance was competing for resources with the application server running on the same box.

Hard to maintain

Any upgrades or patches required for the MySQL server would have to be installed manually by our development team, meaning time spent maintaining a database and not building new features.

Hard to recover from failure

Recovering from a failure to the database or its data volume would require a similar sequence of steps as listed above for expanding the volume and/or increasing the compute capacity.

Enter RDS

Amazon RDS provides an elegant solution to all of the above issues. It is a standalone service designed, primarily, to solve the first two problems on our list: growing and scaling. Resizing it and scaling up its compute resources are each completed with a single API call and minimal downtime. In addition to this, Amazon administers the underlying MySQL infrastructure for you, freeing you of any DBA responsibilities such as patch updates or backups. Restoring from a backup is, again, one command away.

How we moved

Our first step was to perform a dry run of the migration by launching a new, standalone EC2 instance based on the same AMI running our primary app/DB server instance. (With the AWS console, creating a new instance is only one click away.) This allowed us to know we were testing a system that behaved exactly like our production system, while not interfering with our production system. We pushed the latest version of our code to the new instance, and verified that we could hit the app through a browser.

Next, as ruby developers, we decided to use a ruby library to interface with the RDS API. We installed the amazon-ec2 gem locally in our development environment and configured it for use like so:

Once we had an RDS object, we were ready to create a brand new RDS database instance with plenty of space and in the same region as our two EC2 instances:

With the DB instance created, the last remaining piece was to configure its security group to allow network ingress from our EC2 security group:

Our RDS database instance setup was now complete. The last major piece of the dry run was to migrate our data from the local MySQL database to the RDS database and verify that the application was still running. Since our database was under 2 GB, we decided to use the mysqldump utility to create a flat file containing our schema and data, and the mysql tool to import this file into our new DB. The entire process only took a few minutes:

mysqldump old_db -u user -p > inventables.sql

mysql rds_db -u user -p -h [RDS_connection_string] < inventables.sql

With the import complete, we modified our config/database.yml on the test machine to use the new RDS connection string, stopped our local MySQL server, restarted Apache, and verified that the application was still running via a web browser and the rails console. Success!

Production Migration

With the dry run out of the way, converting the live system was a piece of cake. We put up a maintenance page, turned off all cron jobs (which also interact with our database), then repeated the dump/import and database.yml changes in the production environment. Instead of testing the new DB connection via a browser, however, we tested using only the rails console. We also made sure to test a fresh deployment with and without migrations.

Conclusion

The process of moving from a local MySQL database to RDS was quite painless and only took a few hours. We now have a much more robust architecture that will allow us to more easily respond to the growth of our website.

—Written by Jeff