robotmay

I'm a programmer based in the UK. I work primarily in Ruby, but also dabble in JavaScript and Go. I'm now part of the team at Bare Naked!

[robotmay-blog]

I just chucked up a quick one-day-build application for sharing short passages of text. There's no formatting support save plain text. You can sign in with Twitter to post your own items, and every bit of writing has a short URL for easy sharing. Let me know what you think and how I can improve it!

[robotmay-blog]

Just came across this interesting article regarding image manipulation in Go. I found it quite an interesting read as I'm about to start experimenting with image manipulation for a hobby project.

[robotmay-blog]

Scaling down - operating low traffic applications efficiently

I absolutely love learning about operating applications at a large scale. Reading about great (and crazy) solutions developers have invented for handling millions of users a day is really exciting.

But perhaps, like me, your own applications aren't quite there. You might be running at a low scale and financing your project out of your own pocket and working on it in the evenings.

Maybe it's just not popular yet.

Application performance

This article relates primarily to Ruby on Rails but the principles remain the same when applied to most other environments too.

I constantly come across warnings against "premature optimisation". It's sensible to spend more of your time fleshing out the application and adding features, rather than making small gains in page load times in the early days of the project.

But what if we're not just optimising for speed, but for cost efficiency? Or better still; what if we're optimising because we can?

There's little harm in learning and experimenting in performance tweaking on a low-traffic application (unless you're under heavy time constraints) and it's good practice for when you do hit it big. Days Out Near Me (currently my only live, personal app) was totally prematurely optimised; its traffic is low and I probably didn't need to implement streaming requests or heavy caching. But then it hit the front page of Hacker News and nothing bad happened. I'd already eliminated any of the large bottlenecks which could have brought it down when I was most eager for people to be trying it out.

I'm going to detail a few tweaks I've used in the past to help run applications at a low scale.

Caching

It takes very little effort to start implementing caching. Use Memcached and just shovel stuff into it; you'd be surprised at how little really needs to be loaded fresh every time (dependent on the application, obviously). There's a good article in the Rails guides to get you going and I've written before about the benefits of going deeper and using low level caching.

Memcached has a great feature for running on a budget; it will automatically expire the oldest records if it's running out of memory. This makes it very viable to use one of the free options on the Heroku addons when operating at low scale.

Asyncronous page loads

When I first launched Days Out Near Me I had a cron task running once per night to fetch and cache Flickr and DuckDuckGo data for each place and event. That worked fine for a few weeks until I was sending a few thousand requests out each night. I remedied it for a while with a hack; I set random cache timers between 1 and 3 days. That reduced the amount of requests being made each day, but it still wouldn't scale much further without getting me rate limited on those APIs.

Instead I moved all the queries for API data into jQuery.load requests pulling in the relevant areas of the view after the page had loaded. This is a (vastly simplified) technique used by applications like Facebook to improve the perceived speed of the site.

Here's a simple example; the following forms the basis for how a place page loads extra information from DuckDuckGo on DONM (e.g. Castle Drogo):

First we have the method on the model which returns the relevant information from the API for a record. This API data is cached for a few days as it's highly unlikely to change for historical properties:

Then we have a new method in the controller which returns the HTML for that information without a layout:

You need that new action in your routes:

And finally we want to load that content into the page. Here's a very simple example of how you could do that:

This also has the added benefit of failing invisibly for the user if something goes wrong with the API request, such as a timeout. If the ddg_info action throws an error it simply doesn't appear on the page.

Useful services & libraries

Cloudflare is fantastic and will really help your page loading speed performance. It's also free, which makes it even better.

Want to save yourself storage costs in combination with Cloudflare? Use Dragonfly for your file upload needs. Dragonfly generates thumbnails on the fly and relies on a caching proxy to store those generated versions. This means you don't have to store multiple versions of the same image; only the original. The best bit? It works with Cloudflare.

If you want to save some time in view rendering then give Slim a go.

Hosting platform

For the purposes of this section I'll be using the Heroku terminology of web/worker/other processes.

When it comes time to deploying your application you'll no doubt be making the choice between managing your infrastructure yourself, or entrusting that to a Platform as a Service (PaaS) such as Heroku or Engine Yard. The primary resources you'll be concerned with differ between the two setups, though ultimately boil down to the total monetary cost:

On a traditional platform (VPS/dedicated server etc.) processes will cost you memory, CPU, and I/O.

A typical Rails environment rather likes its RAM and will abscond with copious amounts of it, depending on the application. You also don't want to be running so many processes that you max out your CPU or storage allocation.

On Heroku processes will cost you money

At $35 per month per process; you don't want to be running more than you need.

Web processes

Unicorn, as I've mentioned before, will help you serve more requests with fewer processes/dynos. Make sure you check out the original article from railsonfire. For easier reference, here's my Unicorn config:

Background processes

Processing certain tasks in the background can greatly improve the user experience by moving slower, blocking requests (such as sending an email or generating thumbnails) into a queue which is processed separately from the main web request.

Sidekiq

If you're doing quite a bit of background processing you may want to look into Sidekiq. This is a late addition to this section as it has somehow completely passed me by; I only just found out about it via Sam Soffes' blog post about scaling Cheddar. Essentially it's a more efficient background processing library which is compatible with Resque. This should allow you to run fewer workers to handle the same background load.

HireFire

One of the easiest options for scaling Rails workers on Heroku is the HireFire gem (also available as a hosted service). This hooks into your application and automatically launches workers (which will shut down on completion) when there is work to be done. Unfortunately there's currently no support for the Cedar stack in the gem, so if you're not looking to add another fee to your monthly bill (which kind of negates the point of this article) you could instead try the following alternative.

Occasional workers

This example uses Resque as it's the one queue system I know of which supports this out of the box; I'd be happy to add methods for other libraries if you want to fork the example and let me know.

Resque takes an INTERVAL number as an environment option when you start up the rake task. This dictates how often the worker will poll Redis for new jobs. Looking through the source for Resque, however, I found the following in lib/resque/worker.rb:

Notably this line:

break if interval.zero?

So if we pass an interval of 0 to the rake task, the worker will automatically quit once there are no longer any tasks to process:

rake environment resque:work QUEUE=* INTERVAL=0

Great; your workers will now quit whenever they don't have something to do. That's just dandy.

But how is that useful?

Don't run workers; run one-off processes

When operating at low scale you probably aren't sending one email per second, and spending $35 a month (or another chunk of resources) to relieve a little bit of frontend latency is probably not something you're overly happy about. Instead you could run a worker as a one-off process via a cron task (scheduler on Heroku) every 5/10/some minutes and significantly cut your costs. You might want to do this from your own custom task which invokes the Resque task, so that you can check for existing workers before launching more, or log statistics etc.

If you're on your own server you'll want to add that command to your crontab, and for those on Heroku you can read up on how to use the scheduler in the documentation. The whenever gem is a nice way of managing your cron tasks with Ruby, if that tickles your fancy.

And another thing

Some projects are probably not destined for high traffic and amazing success, but equally they aren't so limited in scope that they can be run from a freebie Heroku instance. You have to be willing to put some money into fully realising your idea. Maybe that idea won't work out and you'll have lost money on it, or perhaps you're on to something and your investment of time and money at the start helps you to fully realise your project; either way it's certainly no wasted effort to spend a little extra time optimising your application so that you get the most for your money.

I'm not advocating that everyone spends hours, days, or weeks performance tweaking their application from day one. But why not take it into consideration as you build? Performance tweaks themselves scale well; in low traffic applications they improve the user experience and save you money, and as your app grows in size it will lay the foundations for further improvements and give you a little room to breathe.

What tricks do you know for running an application on a shoe-string budget?

[robotmay-blog]

Having a Go

Right, that's the pun out of my system.

I spent the weekend taking my first steps into Go, which turned out to be a (mostly) great learning experience. It's my first time working with a statically typed language and everything went better than expected. I thought I'd share what I wrote and some of the problems I ran into on the way.

Go is impossible to search for on Google

Probably something many of you have noticed; Go is an absolute pain to search for. I've been using 'golang' as a term but I'm still not sure it brings back as many results as it should do.

The tour is great but makes assumptions about your knowledge

The documentation that everyone points to is the Go tour. It's very good and will teach you pretty much everything you need to know, if you understand it. Coming from a dynamic language like Ruby (and my past experiences being in other dynamic languages) and not having a grounding in computer science meant that I had to spend quite a bit of time researching terms used in the tour. I feel like I've learnt a pretty decent amount but I've spent quite a lot of time asking questions in the #go-nuts IRC channel.

The #go-nuts IRC channel

The guys in the IRC channel are really, really helpful. It's very rare that a question goes unanswered in there, and they've been patient despite my odd questions. After the tour this should probably be your first port of call for help.

Writing my first Go application

I recently signed up for the Network Rail Datafeeds API which features some "real-time" ActiveMQ streams. As I've since discovered; I have no use for this data that I can think of. Instead I'm putting my code below in the hope that it might be useful for others and to get some tips on improvements I can make in future Go applications.

This uses the stompngo library from Guy Allard.

It's a pretty straightforward application: set your access details as environment variables and it connects to the train movements stream and prints the data to your terminal as it receives it. I have no doubt that it could be dramatically improved, so if you have any ideas then please feel free to fork the gist and let me know!

Update

Totally forgot to mention that I've been compiling a list of Go resources using Kippt. If you want to add something to it then get in touch and I'll add you to the collaborators (I'll hopefully turn it into a more easily editable list in future).

[robotmay-blog]

Quick tip: Using an SSH proxy

I've finally found time to get a new blog post written up, which will hopefully go up early next week. In the interim, here's a quick tip I use if I ever need a web proxy (which you may well know of already). This should work on most Unix machines with OpenSSH installed.

The Server

Spin up a VM on the cloud provider of your choice, or use a server you already have available.

Create the proxy connection

Run the following command in your terminal, replacing the user and IP with those of your server:

ssh -ND 9999 [email protected]

Here are the man page explanations for those two options (-N, -D):

-N: Do not execute a remote command. This is useful for just forwarding ports (protocol version 2 only).

-D: [bind_address:]port Specifies a local "dynamic" application-level port forwarding. This works by allocating a socket to listen to port on the local side, optionally bound to the specified bind_address. Whenever a connection is made to this port, the connection is forwarded over the secure channel, and the application protocol is then used to determine where to connect to from the remote machine. Currently the SOCKS4 and SOCKS5 protocols are supported, and ssh will act as a SOCKS server. Only root can forward privileged ports. Dynamic port forwardings can also be specified in the configuration file.

Configure network settings

Now you need to configure your browser/OS to use the SOCKS proxy created. You can manage proxy settings directly in Firefox or Opera, but Chrome usually defers to the system settings. I've provided a couple of examples below, but the general gist is that you need to configure your software to use a SOCKS (v5) proxy on localhost:9999.

Firefox:

In Preferences > Advanced > Network

OS X:

Done!

And that should do it. Check your IP and it should now be that of your server. I often use this in work to check DNS propagation, but it's equally useful for encrypting your traffic when on public networks or for watching BBC iPlayer when outside of the UK (or Hulu outside of the states, etc).

[robotmay-blog]

José Valim just posted a great article which goes into the platform constraints of Rack and how they affected the HTTP streaming implementation in Rails. Well worth a read, especially if your interest in HTTP streaming was piqued by my previous article.

[robotmay-blog]

Forgotten Rails features: HTTP Streaming

The Ruby on Rails community is often fast changing; particularly when it comes to features or libraries. This series of posts will look at some of the components of Rails which aren't talked about much any more. Especially those for which the existing blog posts and screencasts are out of date.

HTTP Streaming/Chunking

I'd be willing to bet that the vast majority of Rails developers have never bothered with the HTTP Streaming support added in Rails 3.1. It's a fairly niche feature and there are a variety of problems (which I'll explain below) which may cause you a bit of suffering should you try to explore it.

HTTP streaming sends the response to the browser in chunks rather than in one go; in Rails this is implemented by sending everything before the yield in your layout first, then sending the rest of the page when the specific view (e.g. "posts/index.html.erb") finishes rendering. It's not for every request, but it can significantly improve the experience for your end user by making the page appear to load faster.

The official blog post explains the benefits of HTTP streaming quite neatly:

Streaming doesn't cut latency, neither it cuts the time a dynamic response needs to be generated. But since the application sends content right away instead of waiting for the whole response to be rendered, the client is able to request assets sooner. In particular, if you flush the head of an HTML document CSS and JavaScript files are going to be fetched in parallel, while the server works on generating content. The consequence is that pages load faster.

Changes from existing examples

If you search for information on streaming support in Rails you'll no doubt come across the relevant Railscast. This is a great intro to using streaming in your application but there are a couple of changes you'll need to make:

There is no longer a method of stream, seen in this example:

Instead you'll need to pass stream: true to your render call. The Railscast illustrates this using the following example:

That will still work fine, however if you're using respond_to & respond_with, you can do the following:

Also note that you'll want to use ActiveRecord's delayed query execution as much as possible (in this case using Project.scoped instead of Project.all) as this will mean that your header will return far faster as the queries are performed during view rendering.

Problems you'll quite likely encounter:

NewRelic

NewRelic, that beau of application monitoring, is going to ruin your day. This was causing me no end of trouble until I stumbled across an excellent blog post from Icelab. The gist being that NewRelic's end user monitoring system is injecting JavaScript in a variety of places and will rain all over your parade; it causes the response to block, rendering streaming pointless. Icelab recommend the following tweak:

Action Caching

I forgot to screenshot what happens when you do this; just don't do it. It prints what I believe to be the chunk byte counts (I'd really like to know if I'm right or wrong here) onto the page. Which will then be cached. Lovely.

HAML

HAML doesn't work with HTTP streaming as of today (2012-05-30). I personally use Slim but ERB will work fine too.

content_for

content_for won't work, as your layout will already have been sent to the browser before the view. Instead you can use provide, as demonstrated in the Rails Streaming API docs.

Other things to note

If you decide to use streaming in your application, it's time to go against everything you've been told in the past few years: put your JavaScript include tags in your head tag (not at the bottom just inside the body tag) so that they're loaded by the browser while the rest of the chunks fly in.

As the RailsCast pointed out, you'll be wanting to use Unicorn (or another server that supports streaming). For conciseness, and so that hopefully others can get up and running quickly, here is my config file assembled from various places (though most particularly railsonfire) which I use for Days Out Near Me on Heroku:

Note the references to Dalli which I lifted from a recent commit. I'll hopefully go into more depth with how Unicorn has improved the application performance in the future.

Example use case

I recently switched how Days Out Near Me calculates the nearest places to you from a straightforward geodistance to using a travel time API (more on this soon). This resulted in a lot of performance tweaking to get the search results (e.g. searching for days out near Liverpool) to continue displaying at a reasonable speed (as it now uses up to 3 different services in the action). The adjustments are far from complete but the first step was to use HTTP streaming to improve how fast the page feels. Streaming is only used on the homepage and the search results page currently, as those most benefit from it. However I can see it being useful to roll out on individual place & event pages as those are often the entry points to the site for many users.

Feedback

I'd love any feedback or other suggestions/problems you've come across while using HTTP streaming. And if anyone knows of a better theme for programming on Tumblr then I'd be very interested.

[robotmay-blog]

Looks like my first contribution to [DuckDuckHack](http://duckduckhack.com/) just went live. It's a pretty simple (and my first) bit of Perl which will spell out your query using the NATO phonetic alphabet. You need never struggle when ordering takeaway again!

[robotmay-blog]

I recently released an [OmniAuth](https://github.com/intridea/omniauth) strategy for [Harvest](http://www.getharvest.com). It's pretty basic so far, though I'll be improving it as we use it in an upcoming project. I'd appreciate any feedback and bug reports!

[robotmay-blog]

Everyone should be using low level caching in Rails

Low level caching is very easy to get started with in Rails 3, but it seems to be missing from [the official guides](http://guides.rubyonrails.org). I personally use it all the time to help reduce the number of queries or API calls I'm making. [Heroku has a pretty good explanation](https://devcenter.heroku.com/articles/caching-strategies#low_level_caching) which, if you're so far unfamiliar with low-level caching, is a good place to start. Why should you use low level caching, and what should you use it for? Maybe you have some data which you need regularly over multiple pages, e.g. a list of categories for your blog. You might want to display them on every page but they're not going to need to be entirely up to date for every request. You could stick this in controllers/application_controller.rb: If you haven't used this style of caching before, note that I'm calling `.all` on the Category query. In Rails 3 ActiveRecord now uses lazy loading; it doesn't perform the query until you try to use it. This is great for fragment caching in your views as it will prevent that query being called if a fragment cache already exists. However, if you're using low level caching you'll want to perform that query immediately so that the results, rather than just the query object you've built, are stored in the cache. The mainstay of low level caching in Rails is [`Rails.cache.fetch`](http://api.rubyonrails.org/classes/ActiveSupport/Cache/Store.html#method-i-fetch). There's something you need to know before you end up in a spiral of frustration; by default **this will cache in your development environment**. To remedy this, add this line to your config/environments/development.rb: `config.cache_store = :null_store` `Rails.cache.fetch` takes 3 arguments; the cache key, an options hash, and a block. It first tries to read a valid cache record based on the key; if that key exists and hasn't expired it will return the value from the cache. If it can't find a valid record it instead takes the return value from the block and stores it in the cache with your specified key. It's well worth [reading the API docs for this method](http://api.rubyonrails.org/classes/ActiveSupport/Cache/Store.html#method-i-fetch), as it explains all the options you can pass to it. I use low level caching most for API calls. On [Days Out Near Me](http://daysoutnearme.com) I use the [Flickr API](http://www.flickr.com/services/api/) to find photographs of places. To do this I'm using the [Flickraw gem](http://hanklords.github.com/flickraw/): This searches for a Flickr place based on the latitude & longitude of the one in my database. If it finds any matching places it returns the first entry, as that's most likely to be the correct one. Unless we're dealing with [a moving castle](https://en.wikipedia.org/wiki/Howl%27s_Moving_Castle) then it's pretty safe to say that this place is going to be correct for quite a while, so it's being cached for 7 days. It could however change if we update our record, or if someone adds it as a new place on Flickr where it was previously missing. By using `self.updated_at` inside the cache key we ensure that the cache is automatically invalidated if we update the place (which could include new co-ordinates). Caching works best in layers. This is [nicely illustrated by 37signals](http://37signals.com/svn/posts/3112-how-basecamp-next-got-to-be-so-damn-fast-without-using-much-client-side-ui); don't just use one type of caching, but instead layer one on top of another. You'll end up with the same data cached in multiple places, but when the cache expires you won't be hit with a totally uncached page load. Some of the pages of [Days Out Near Me](http://daysoutnearme.com) would have pretty horrific response rates if I wasn't using a variety of tricks to reduce how much time it spends in the database and API calls. I'll be posting a few more of these tweaks in the near future. **Edit:** This article makes the assumption that you're using memcached or an equivalent (i.e. Redis) for your cache store. I'd highly recommend using the [Dalli gem](https://github.com/mperham/dalli) with memcached. [Thanks to wavesplash](http://news.ycombinator.com/item?id=3981917) for pointing out that I'd missed this section out. _As this is essentially my first blog post, I would be very grateful for any critique or tips you have._

[robotmay-blog]

These are the crawl stats for the last week for [Days Out Near Me](http://daysoutnearme.com). I've submitted the same sitemap to both sites. I knew the gap was quite large, but almost 3000 indexed pages difference? This is why we're all still reliant on Google, no matter how much I want to use [DuckDuckGo](http://duckduckgo.com) full time.