Where Is Joel

This is where I shall share my thoughts. Probably mostly software related. Most likely really.

[whereisjoel-blog]

Dealing with all the pesky nils in hashes

I don't think I've ever worked in a rails project where I didn't see something like the following:

https://gist.github.com/joeljackson/6102232

At some point everyone gets annoyed by some nil that sometimes shows up. TBH, the best way to deal with this is probably to go upstream and decide why this is happening, but we're all sometimes lazy. I figure I might as well throw my own solution into the hat.

The idea is make a Hash-y object that will respond the same as the object you are returning, but will just return another instance of the class instead of nil. It should delegate methods so we can still check #blank? and #present? and whatever else.

Examples:

https://gist.github.com/joeljackson/6102249

And now for the code:

https://gist.github.com/joeljackson/6102235

You can create a NullHash by passing in a hash, and expect the returned object to behave as it should, with one notable exception. You can't make ruby objects to behave falsy, so if you need a falsy object terminate the NullHash.

Go forth and enjoy!

[whereisjoel-blog]

Protip: Installing ruby debugger with rvm

This trips me up every time.

Let's say you have ruby 1.9.3-p448 and `debugger` is in you Gemfile.

Quite possibly the whole bundle process chokes at debugger-linecache, particularly with a newer rvm and an older debugger gem.

Maybe the error looks something like this: 

https://gist.github.com/joeljackson/5892804

What to do, what to do?

Simply:

`gem install debugger-linecache -- --with-ruby-include=/Users/joel/.rvm/src/ruby-1.9.3-p448/`

Now go forth and debug!!

[whereisjoel-blog]

Benchmarking ruby 2.0's lazy evaluation

So you were all probably pretty pumped about Ruby 2.0's lazy enumerators, right? Right? Wait, what?

Okay, start from the start. We all like writing using our enumerator methods. Chaining them together for nice pretty code like this:

https://gist.github.com/joeljackson/5769608

Very nice. ... Wait...  We're actually looping over the array 3 times. That's suspicious. Does it matter? Aren't we actually doing the same number of operations if we loop over an array once and do three operations each time, or three times and do one operation each time?

This is all very confusing, but there is inherent overhead in looping over an array, even if we do nothing. Lets take a test program, that iterates over a long array doing nothing, does something, iterates twice over an array doing nothing, twice doing something and once doing two things:

https://gist.github.com/joeljackson/5769629

What do you think the speeds of the various tests will be?   Here are the results:

https://gist.github.com/joeljackson/5769689

As it turns out it's significantly more expensive to loop over an array a second time doing nothing than is to add an extra operation inside of our first loop. How is lazy evaluation supposed to help us with this? Well, optimally when we do something like map, instead of just looping twice, we can loop once perform both map operations in the single loop. This is accomplished by evaluating results as needed. For example, check out the following sample program and it's output:

https://gist.github.com/joeljackson/5769715

It loops over an array, doing a map then a select, then does the same thing lazy. This is the output:

https://gist.github.com/joeljackson/5769729

Now *that* looks much more like what we want. So, how much awesomer and faster will this be than the old way?! Here is some code to test this out:

https://gist.github.com/joeljackson/5769754

Once we map then select non-lazily, once we do it lazily then don't force any evaluation, lastly we do it lazily then force evaluation. Here are the results:

https://gist.github.com/joeljackson/5769764

Holy crap, that's slow!! So basically for right now we might as well just keep looping over arrays over and over and over and eating the performance hit. But I don't believe that there is no way this can't be fixed, and I can't wait until it is!

And in the meantime there is one new thing we can do with lazy eval, looping over infinite or really long lists and just evaluating the results we need:

https://gist.github.com/joeljackson/5769779

So, that's kind of cool for now.

Until next time!

(Worth noting, code for this blog post is here.)

[whereisjoel-blog]

Promises/A and Ruby

For the last little while promises have been the latest hotness in the world of javascript. Most of us who love our ruby, however, remain un-baptised in the way of promises, so I'll go over them briefly.

Consider how we like to write code:

https://gist.github.com/joeljackson/5559571

What's nice about this code?

1) It reads from top to bottom. Kind of how we expect things to happen based on things we're used to like... time, books etc.

2) It's indentation is sane. Things that are meant to happen in the same logical context happen after each other at the same indentation making it very obvious.

3) All the error handling happens in one place at the end. This means what the non-error related code isn't cluttered with code that deals with errors, making it's functionality more obvious. It also means we don't have to hunt to find out how to deal with error conditions.

This all starts to fall apart when we need to deal with code that happens asynchronously. I have an example project here of what I mean.  In the example project a simple web server serves random numbers, and has a url that can add numbers. The client grabs two random numbers from the web server, then calls the web server to add them up. There is also the potential for the web server to randomly fail. (Yes, it's contrived, but it will get the point across.)

Let's take a quick look at what the server side looks like

https://gist.github.com/joeljackson/5722185

Simply, we have two endpoints that serve up random numbers, one that adds numbers and an index because the same thing I'm about to demo in ruby exists in javascript. Also every endpoint has a 80% chance of failure, so getting two random numbers and a sum has a 1 - (0.8 * 0.8 * 0.8) = 0.47% chance of failing, or roughly half.

This is what nice synchronous client side code to accomplish our desired task looks like

https://gist.github.com/joeljackson/5722296

That's pretty. Everything happens nice and linearly. Exception handling is all on one place. Problems? We have to do three http requests one after another. During the time those http requests are happening the thread is completely blocked and really we could be doing the first two at the same time to increase performance.

Let's take a look then at what a normal asynchronous implementation of this might look like

https://gist.github.com/joeljackson/5722422

That solves the problem of blocking, but the code really leaves a bad taste. There is all sorts of mucky indentation. Error handling is sporadic and littered all over the codebase. There is a spaghetti of callbacks. This is  why we have promises!

A quick rundown of what a promise is. A promise is returned by an asynchronous method as am object that implements "then". "Then" takes two parameters, a callback and an errback and always returns a new promise. Anything returned from within the callback will then be passed on to the next then in the chain, or if it's a promise, the result of the promise will be passed in. Lastly, any errors will be passed down the "then" chain and given to the last errback in the chain

So, we can write asynchronous code like this:

one.then{two}.then{three}.then{four},{error}

Now, let's take a look at how promises clean up the code. This is the same method written asynchronously using promises

https://gist.github.com/joeljackson/5722487

Now, that is much nicer! All error handling is in one location, the indentation is totally sane. Sweet. Now...  let's imagine that we want to do the first two requests at once, let's combine the first two requests into one promise so they can both happen at the same time.

https://gist.github.com/joeljackson/5722530

Our code is nice and clean. The first two requests happen simultaneously, all exception handling occurs in one place. Win!!!

Go check out em-promise and em-http-promise and clean up  your code!  This same paradigm can also make writing beautiful async-rack and async-sinatra a breeze, but I'll write about that another time.

[whereisjoel-blog]

Inheritance vs. Modules vs. Composition (In the context of Rails)

Anyone tried out Rails 4? All sorts of cool things. Turbolinks. HStore. Russian Doll Caching. But while you were busy hyperventilating over whether turbolinks would magically convert your once cutting edge web 2.0 app into a, now much more de mode, single page app, the Rails team snuck one weird little change in under our noses. They added a "models/concerns" and a "controllers/concerns" directory to encourage your pretty little faces to do a little refactoring. 

1) What on earth is a concern? See here. It's basically a module with some funny syntax that does some fancy, but most likely unnecessary,  dependency management.

2) Why do you care? Well, this is as good of time as any to talk about where code belongs in our Rails apps and when we should include modules.

So you're working on a Rails app. As you work you repeat the Rails mantra to yourself over and over, "Fat Models, Skinny Controllers" and you've even done a pretty good job of sticking to it. Well done! Yet...  Things are starting to get a little bit messy. Your models are getting too fat, I'm talking John Pinette fat, and you are beginning to suspect you've been lied to and there is a little bit more to this whole software design thing.

Fear not! Inheritance, inclusion and composition are here to save you! And yes, I know, this has been written about before. The thing is that, for whatever reason, the examples used always involve dogs, cats, Pomeranians and your cousin. In the real world I have never actually written "class Dog". This is a real world example of how to use these concepts. Real world, like straight from the street.

Lets pretend you're building a website called tumblr_ripoff.com. As you may have guessed, it's a ripoff of tumblr. This website has the following requirements

1) Users can post picture or text posts

2) A certain subset of users are notified when a post is added or a comment is made on a post

3) The notifications are stored with relation to the post and the users who were notified should be findable by post

To this end you have created the following ActiveRecord class:

https://gist.github.com/joeljackson/5003095

TBH, it's not half bad, but the road to hell is filled with good intentions

Lets look at the problems with this class

1) You probably also have a class called text which belongs to user (And might do some similar blog posty things) which means repeated code. Any time you change one you have to change the other.

2) Comments also have notifications, so we'll need the notification code there too, once again repeated code that needs to exist in two places.

3) Why does a photo know anything at all about how to send notifications? It's an ActiveRecord class, it's responsibility is to expose data, allow saving of data and define relationships. (God, the poor thing has enough to do already.)

So let's fix this up, each of these problems is solvable by the respective technique mentioned above.

1) Let's create a post class that photo can inherit from to share logic with other types of posts. (Aka. text)

2) Let's create a notifiable module so that it can be mixed into non-post type classes which need notifications. (Aka. comments)

3) Lastly let's make a class that knows how to make notifiable objects then send the notification.

And (drum roll please.)

https://gist.github.com/joeljackson/5003155

 Now...  What have we learned?

When to use inheritance: Pretty rarely to be honest. If the bulk of the responsibility of two classes are the same, and I mean like really most, then it should inherit.

When to use inclusion: Well, this is where "concerns" come in. Just remember, moving code to another file does help share code between your classes, and it does make the class look smaller, BUT once its included in the class, in reality the class is not any smaller. Include code if it falls within the responsibility of a class, and only then and if it needs to be shared between two files.

When to use composition: In short, try and prefer composition. A few strategies? Do you have methods on your models that modify them? Consider moving them into classes that change the models, particularly if they involve multiple classes. Do you have methods all throughout your models to display things on the screen? Move them into decorators. Do you have methods on your models to compare values? Move the values into value classes.

At some point later I'll talk about these and maybe other refactoring methods. Until then take this away from this post, keep your classes small, keep your methods small, and favor composition over inclusion.

Now go forth, refactor and build!

[whereisjoel-blog]

Debugger in ruby 2.0 got you down?

I have the debugger blues too.

What's the problem?  In Ruby 2.0 next acts like step, so you have to step through every single line of inconsequential code in your application. Also backtrace has a disturbing tendency to crash your app.

Check this gem out to get you through these difficult times. It even has pry bindings. Sometimes I've found it crashes inexplicably, so maybe keep debugger in your app too. They're bound to get it working soon

[whereisjoel-blog]

Custom Classes and Array#uniq

So you have a nifty little value class or something in your ruby app. 

Let's say: 

https://gist.github.com/joeljackson/5602698

It's been serving you pretty well, you can write all kinds of neat things in your code that would otherwise have been much uglier like:

https://gist.github.com/joeljackson/5602710

And you're generally very happy with your life. Then you go and do something like use Array#uniq. You'd expect it to work, because naturally it would be implemented using "==".

But...

https://gist.github.com/joeljackson/5602722

Well, that doesn't seem right, so we visit the docs.

From the docs: "It compares values using their hash and eql? methods for efficiency."

Well, crap. Now what? I'll tell you what: 

https://gist.github.com/joeljackson/5602735

That's what. And *now* your precious Array#uniq will work.

Enjoy!

[whereisjoel-blog]

Asynchronous IO in Rack Middleware

At Xfire we've recently built a matchmaking service for users of our Battleground platform. The Matchmaker is a little Sinatra server that gets a request, creates a challenge, and tells the Xfire client about the challenge id.

For authentication, the client sends some parameters to the Matchmaker, and the Matchmaker calls an internal API call to make sure the user is who they say they are. I cooked up some simple middleware that performs auth, and by the time any request reached sinatra it either had the user_id or not depending out whether the user was authed.

The problem? During the HTTP request to the API, the server would sit around blocking doing nothing. (Okay, blah blah, threadpool, blah blah, but you get the point.)

Time for some fun with Asynchronous rack. Take the following synchronous Rack application:

https://gist.github.com/joeljackson/5519157

The application has two URLs it handles

1) "/sleep_time?sleep_time=#{some_time}" 

2) Everything else

The first will sleep for a certain amount of seconds and return. For every other URL, the app does an http request to /sleep_time with an amount of time, waits for the response then says "hello world". If we tell thin (the webserver of choice for this example) to not use a threadpool for it's requests we can never even get a response because the request to "/sleep_time" will block waiting for the first request to finish it's IO. With two threads it can handle one request at a time, with 4 it can handle 2 requests at a time etc. Let's do better and rewrite this little rack app to do things asynchronously so it never blocks it's current thread.

https://gist.github.com/joeljackson/5519207

Now, when we do a request, it does a non blocking IO call to /sleep_time, then makes way for the next request, ALL ON THE SAME THREAD!! :-) This little guy will handle pretty much unlimited (okay, there's a limit, but it's not due to threads blocking) requests at the same time and smile the entire time.

What did we do? First each request that needed to be async did a "throw :async" instead of returning. (You can also return a status code of -1, or 0, but that's not consistent across webservers, so just use throw).

THEN, We did a non blocking IO call or non blocking sleep using EventMachine and called the async callback given to us by rack. Voila! Non blocking IO in your Rack app.

Now, I need to stop procrastinating and get back to work. If you're interested in seeing how we can build something a bit more complicated, check this out, but really this simple example should get you started. 

Later!

(BTW. If you want to get the examples working to play with, then install thin and em-http-request and you'll be good to go!)

EDIT:

It occurs to me that this post might actually suggest a very bad habit to those looking to it to actually create middleware on a thin/eventmachine stack. The callback for the HTTP request will occur on the main reactor loop, which is fine if you are running thin unthreaded. Say however you are running threaded thin, and your app takes 500ms to do something very important like find all the latest in trendy cardigans for men. The reactor loop and all IO will be blocked for the entire 500ms, preventing other requests. It would behoove the prudent rubyist to actually call their app within a defer block so that it was executed within EventMachine's threadpool. An actual middleware implementation might look like this:

https://gist.github.com/joeljackson/5519815

[whereisjoel-blog]

Supersize Me Too (Two McDoubles a day keeps the fat away)

Last week, on a busy day at work, I decided I was in the mood for a fast cheap lunch. I grabbed 2 McDoubles at McDonalds, hid them lest it be noticed I was eating something other than kale in the health crazy city of Santa Monica, and hastened back to the office.

I expected to get back to work, sit down at my desk and continue my work, but instead found myself lambasted with accusations of being condemned to a life of obesity. I objected "A few McDoubles here and there is not a life sentence to being fat", which according to my accuser was just the attitude that keeps America fat.

Now, a McDouble is certainly not what I would describe as the paragon of healthy eating. Being very high in Sodium and Fat I understood that it's not like my heart was gleefully sitting around waiting to pen me a thank you note. However at 370 calories I was pretty sure I could eat two of these things every day and as long as I was careful about the rest of my diet not gain any weight. 

Being a man of conviction I decided to put my money where my mouth was and bet I could consume two McDoubles at lunch every work day this month. I'll eat a normal diet of 2000-2500 calories per day with 800 of those calories coming from the vilified McDonald's sandwich. I am also prepared to be completely amazed if during this time my body decides McDonalds calories are somehow magically more fattening than normal calories.

What I'm eating will be completely documented here: http://www.myfitnesspal.com/food/diary/jacksonjoel

I'm starting out weighing 198.3 lbs and looking like this after a shower in the morning taking a selfie like a 15 year old (no flexies).

Lastly, here I am consuming the initial McDouble to kick this thing off, in what I like to think is a terrible photo of me:

I'll keep everyone posted.

[whereisjoel-blog]

Ruby Code Reloading, Threads and You

As part of our ongoing work on Xfire Battleground we built a program called "The Judge" that keeps track of what users are doing in game and awards and takes points away depending on various in-game outcomes.  

The Judge runs many threads doing a wide range on things with a wide range of importance. One source of irritation for me was that after every code change I had to turn off the judge and turn it back on to test whether what I had done worked. What this meant was a 5-6 second wait on my macbook, and by the time it was done launching I had inevitably become bored and was already checking in on what Kim Kardashian ate for lunch.

To alleviate this I wanted to build a simple code reloader, but of course time was limited. Step 1) build a quick test.  Here's what I wrote:

https://gist.github.com/joeljackson/5464335

https://gist.github.com/joeljackson/5464342

Now, forgiving my ridiculous variable names, if you run main, and then change the text in loop, any ideas what will print out?

Later!

1 Day Later:

Okay, so... I changed the above to make the code a little more legible in case someone actually wants to read it. If you ran the code you would have discovered that everything in print_me method updated when we reloaded the code, but nothing inside the loop in run2 updated. Pretty much as we expected. 

When we overwrite any method and it gets called and it gets looked up, the new method is called, exactly as we desire. So, it's not perfect, but it will do. I'll update this later with a quick gist of what the final solution ended up being just in case anyone else is interested in using it.

(BTW. This abuse of ruby monkey patching is not really require friendly. Also, instance and class variables won't actually reload, it's not going to work AT ALL if you change the inheritance of a class. yada yada. YMMV, but for most things it should get you most of the way. Oh, remember, methods you remove from a class are NOT going away.)

[whereisjoel-blog]

Opinionated Software & U & Rails

I love opinionated software. I love it. For most of us building end user products it's much better to just rely on people to build technology while we provide the business logic glue. No one has the time or resources to develop both technology and product. On top of this, aside from the fact that we rely on stacks of technology that would take teams of good engineers years to reconstruct themselves, it also exposes us to the good ideas that other groups of people have and allows us to incorporate that in our technology.

To that point I encourage anyone who use rails to try jazz_hands. It includes a collection of gems I add to every project I work on. Do it. Just do it. Worst case scenario... nothing. Seriously, worst case scenario you don't use is, and your life won't change at all.

[whereisjoel-blog]

And now I feel like espresso

This is what Starbucks' espresso webpage looks like:

Interesting to note that every picture looks like it's shot using a tilt-shift sort of angle and blur. Tilt shift photography is a technique often used to make objects in focus appear smaller than the actually are, often to the point of making large objects look toy sized.

I wonder if they are shrinking the appearance of their whip cream topped sugar-fat-coffee drinks so we are subconsciously less likely to associate them with a pile of calories?

Now excuse me while I go partake in a Venti-Extra Whip Mocha.

[whereisjoel-blog]

The Less Popular 'Maybe Me Call'

[whereisjoel-blog]

Voronoi Diagrams (AKA. Pretty pictures.)

A Voronoi diagram is a diagram wherein regions are laid out such that given some set of generator points, every point within a given region is closer to it's own generator point than any other generator point.

A friend of mine was playing with making a generator in js here. Go ahead and check it out. It will freeze your browser for a sec while it figures things out, don't panic.  If you don't want to look it looks like this: 

Now the real fun starts. The distance function he used is linear distance as follows:

https://gist.github.com/joeljackson/5315990

But voronoi diagrams can really benefit from unconventional distance functions. For example here is a quartic one:

https://gist.github.com/joeljackson/5316007

And here are some cool examples of what happens when you mess with distance functions.

Manhattan distance:

The above mentioned quartic distance function:

And a really crazy sin distance:

I've made a jsfiddle of his code to play with here.  Have fun!

Thanks Adam for the fun!

[whereisjoel-blog]

The Omni-Awful Refactor

Anyone who has spent a significant amount of time working with Ruby on Rails has at some point encountered OmniAuth. OmniAuth exposes Authentication strategies for various services as a singular interface, it can plug into LDAP, OAuth, OpenID etc. If you haven't heard of it before, check it out here or check out the RailsCast here.

Anyone who has worked with OmniAuth over multiple providers for a large project is probably also aware that the code used to deal with it can very quickly degenerate into a large steaming pile of shit.  The project page shows the following example: 

https://gist.github.com/joeljackson/5255644

This simple piece of code actually hides a large amount of complexity.  Let's take a close look at the railscast which shares this piece of code:

https://gist.github.com/joeljackson/5257081

The Sessions controller create action has three responsibilities.

1)  Find the correct user and sign them in. (Via omniauth, possibly password, etc.)

2) Create users from various external services

3) Associate existing user accounts with external services.

This is further complicated by the fact that for each service we might want to do slightly different things. For example with some services we might want to collect extra information that does not come from omniauth. (For Linked In for example you might want to build a resume, or whatever.)  The code quickly spirals into something like this:

https://gist.github.com/joeljackson/5257124

And now we're quickly on the path to disaster. Flushed out that code is hard to read, hard to test and generally smelly.

I'm not proud to admit we went pretty far down that path at Xfire before finally biting the bullet and refactoring. We quickly decided that from an overall perspective the sessions#create action should have the following structure:

https://gist.github.com/joeljackson/5257171

Each chunk would have to be testable. We wanted to abide by the single responsibility principle. We wanted to not clutter the controller. We decided on an architecture where each block would be executed as a strategy, a generic class would run strategies, and for each type of strategy a class would know how to pick the correct strategy based on the request as well as the parameters to pass the strategy. The completed sessions#create method looks as follows:

https://gist.github.com/joeljackson/5257236

The auto_login method is basically the same as the one used in Sorcery if you're interested. It's included in ApplicationController via a module. The StrategyRunner class looks like this:

https://gist.github.com/joeljackson/5257262

It gets the appropriate strategy finder class for the strategy we're looking for, passes it the parameters to figure out the strategy, gets the strategy class back, runs it with the parameters from the strategy finding class.

A strategy finding class might look like this:

https://gist.github.com/joeljackson/5257336

In this case it knows which authentication strategy to pick based on the request, and what parameters to feed it

Lastly an actual strategy class is really simple and need only implement one method, run. It looks like this:

https://gist.github.com/joeljackson/5257378

And we're done. Every component now has one single responsibility so while developing it's easy to understand the code. The code is easy to test. It can be used elsewhere in the application if need be. I'm pretty happy with it. Thoughts? Concerns? Points that make you furious?

[whereisjoel-blog]

Fragment Cacheing and Cache Keys

Not that the following happened to me or anything...

So, you use Ruby on Rails. And you have a particularly slow partial that gets called on all the time. No problemo right? Let's just wrap that guy in some cacheing and move on. You do something like this:

https://gist.github.com/4984376

Great, your site is now running way faster. Now it's time to update that partial. Boom, update the partial! Wait... My site looks exactly the same. Okay, obviously cache takes two hashes. That code really should have looked like this:

https://gist.github.com/4984428

Lesson learned, I'll just fix that right up and... Right, that partial is getting hit all the time, that's why you put it in the cache to begin with. So it's definitely not falling out of cache. Alright, "how do I fix this"?

So, first thing, I guess you try using expire_fragment in the console, but of course this all depends on the scope it's called in so that's not working out.  What to do? What to do?  Well, let me tell you, you have come to the right place.

Do this in the console: (Thank me later.)

https://gist.github.com/joeljackson/4984474

Of course if you just want to expire the fragment and you didn't do the stupid expires in thing, just ditch that at the end.

Cheers!!

Joel

[whereisjoel-blog]

Better Errors in Rails (and Sinatra and Rack)

To sum up what I'm about to write for people who are either lazy or just don't like reading: "Add the gem better_errors to your project. Your life will instantly become better."

Ok, so you've got a Rails project. Or you have a Sinatra project. Or you're just working away in Rack because you're just that 1337.

Inevitably at some point something goes wrong. You get an error somewhere. If you've been working in Rails you probably have traditionally considered yourself to be pretty lucky. You get a nice little stack trace, it tells you where things went wrong:

  Thing is, then you look at your code, and it's just not quite obvious what the issue was. So maybe you add a "debugger" statement to check out what was going on. Maybe you like pry, or you're more into RubyMine, so you click and add the brea...  Stop right there, WAY too much work.  

Just add these two lines to your Gemfile:

https://gist.github.com/4942831

And NOW your error page looks like this:

You can run code live in the browser to see what was going on, check all the variables all the way up the Stack Trace! Holy Crap.

More importantly why are you still reading this? Just go add it to your project right now. Do it already. Go. Chop chop!