On and On

Lessons learnt on the way to becoming a better web developer and human bean. Lots of terrible jokes included.

[murphydbuffalo]

Why do I need such a large sample size for my A/B test?

Running A/B tests is about finding out with some degree of confidence which of two versions of some thing is better. The canonical example would be running a test on the sign up form of your website. At present some percentage of users who see that page sign up or “convert”, and you’d like to see if making some change to the text or look of the page will make a greater percentage of users convert.

You can determine if that’s the case by running a hypothesis test comparing the proportion of users who convert when shown the original version (let’s call this p1) and proportion who convert when shown the new version (let’s call this p2). The null hypothesis is typically something like p1 - p2 = 0, or that there is no real difference between the two versions. And the alternate hypothesis is therefore something like p1 - p2 != 0 or p2 > p1, meaning that there is in fact some difference between the two versions. This difference between the two proportions is called the effect size, and it is often measured in standard deviations. For example if version A shows a conversion rate of 5%, version B shows a rate of 6%, and the standard deviation is 1% then the effect size is 1 standard deviation.

As you probably recall from statistics class back in the day, you must select a significance level for your test. If you see an effect size that is more standard deviations away from the mean than your significance level then you have a “significant” result.

All of this is well and good, but if you’ve looked into recommended sample sizes for running your experiment you’ve likely been surprised by how many examples are required: often many thousands for each version.

What exactly do we gain by using these recommended sample sizes as opposed to running our test until we’ve found a significant result? Quite a lot it turns out. 

The Goal

Looking at your A/B tests only through the lens of significance gives you an incomplete and sometimes misleading picture. I would argue that instead of being concerned with significance, we should be concerned with the likelihood that the conclusions we draw from our test are true, and that those conclusions signify something important to us.

This goal can be broken down into more digestible chunks. First, what does it mean to be concerned with the likelihood of our results being true? This comes down to the fact that there are four possibilities when we run our test:

1. The null hypothesis is false and we reject it. A true positive.

2. The null hypothesis is true and we reject it. A false positive.

3. The null hypothesis is false and we fail to reject it. A false negative.

4. The null hypothesis is true and we fail to reject it. A true negative.

We want to be reasonably confident that our results fall into the true conclusions covered by numbers 1 and 4, and not the false conclusions of numbers 2 and 3.

By adjusting our significance level we can directly control for the likelihood of false positives. A significance level of 5% means that assuming our null hypothesis is true there is a 5% chance that by pure luck we will see a result as or more extreme than that significance level.

That covers false positives, but what about the risk of false negatives? That’s another importance piece of our quest for truth. Power in statistics is the probability of your hypothesis test detecting a positive result when it is present. Said another way it’s the probability of not having a false negative.

So in order to effectively pursue true results we need to be concerned with both the significance level and power.

What about the second part of our goal, conclusions that signify something important to us? This addresses the fact that you can find statistically significant results that are utterly uninteresting and not worth acting on. That is, you could run an A/B test and find a significant result that says “version B is 0.0001% more effective than version A”. Very likely, although it depends on your domain, this difference is meaningless and it wouldn’t be worth the time and effort to run the test and then replace option A with option B.

Crucially, such uninteresting results are more likely to be found as your sample size increases. This is because the larger the sample size the lower the standard deviation of the sampling distribution. Think back to our example from the beginning of the post where we had an effect size of 1% or 1 standard deviation (because our standard deviation was also 1%). If the sample size goes way up, shrinking the standard deviation to 0.1%, then the same results would show an effect size of 10 standard deviations. This would be much, much farther from the mean of the distribution and would almost certainly qualify as a significant result.

The point is that a large enough sample size can make any effect size significant. So it’s important that we define ahead of time the smallest effect size we deem to be meaningful to us. In the context of A/B testing this might mean the smallest effect size that would make it worth our while to wait for our test results and deploy a new version of the feature.

We are going to use the figure we come up with here as something called the Minimum Detectable Effect (MDE) in our sample size calculations. There will be more on MDE towards the end of the post for those who are curious, but it’s not necessary to understand it in detail to grasp what A/B testing sample size calculations should do for you.

What Your Sample Size Tells You

Taken together, what does all of this mean? We want to have a level of confidence that any conclusions we draw are true. We specify that level of confidence in terms of significance level to deal with the probability of false positives and power to deal with the probability of false negatives. We also specify the smallest effect size that is meaningful to us in order to avoid running a test only to find out that we don’t care about its results.

Given all of this we can solve for the smallest sample size that will satisfy all of those parameters. This will tell us “you need X trials per version for your experiment to have this level of significance level and power for an effect size at least as big as your MDE”. If your effect size is bigger than that MDE, awesome! You’ll have an even greater chance of detecting it than your specified level of power.

Somewhat counter-intuitively, you do not want to keep running your test beyond this sample size. Remember, that this means you will increase the risk of detecting as significant results smaller than your MDE, which by your own definition is the smallest effect size you would care to notice.

Equally important is avoiding the opposite temptation: not running your test until you’ve achieved the calculated sample size, and instead stopping the test as soon as you’ve achieved significance. Evan Miller does a wonderful job explaining why you shouldn’t do this, but the short answer is that it biases you towards thinking you have significant results when really you don’t. What if after 500 trials the result is deemed significant, but after 1500 it turns out the result isn’t after all? If you had stopped early you would have drawn the incorrect conclusion about your results.

Whew! That’s a lot and certainly more complicated than simply checking if your result is significant or not. But it is also an approach that will get you results you actually care about with a level of reliability you can be comfortable with.

Postscript - Explaining Minimum Detectable Effect

The MDE is the smallest effect size needed for a test to have at least a certain level of power, given a particular significance level. For example if you know that you want your test to have an significance level of 5% and a power of 80% you could calculate an MDE which would tell you “if I see an effect size equal to or greater than our MDE we will have at least an 80% chance of detecting a positive result given our 5% significance level”.

Jonathan Leirer has a great article explaining this better than I can. But the heart of the concept is that for a positive result the probability distribution of the null hypothesis (the “null distribution”) is not true, rather some alternate hypothesis with its own probability distribution is true. Further, the null distribution and the alternate distribution will be some distance apart, and may have some overlap, thus the possibility of false positives and negatives.

So, if we want a power of 80% the MDE describes how far from our significance level (which is on the null distribution) the alternate distribution mean must be for 80% of the alternate distribution to fit in the space up to that significance level. Said another way, if the alternate hypothesis is true, and 80% of the alternate hypothesis’s probability distribution falls past our significance level, we have an 80% chance of detecting positive results given a significant result. It’s worth remembering here that ”significant” simply means that our observed effect lies beyond our set level of significance level on the probability distribution. A picture really is worth about 1,000,000 words here, so read Jonathan’s article to see a visual of how this works.

[murphydbuffalo]

A Tale of Two Concurrency Models

At Bonusly I work in a Ruby on Rails codebase, but in my free time I like to explore functional programming with Elixir. One of the reasons Elixir is so interesting to me is because it compiles to bytecode for the Erlang virtual machine, BEAM, and BEAM (Bjorn’s Erlang Abstract Machine) is a big deal.

What I’d like to do in this post is compare and contrast the concurrency model typically provided by object-oriented languages like Ruby and Java, with the model provided by the functional languages Elixir and Erlang.

If you’re interested in a deep dive into this topic I’d definitely recommend picking up Elixir in Action by Saša Jurić. It’s a fantastic book, and it covers all of the material in this post in greater depth.

High Scalability

The Erlang VM was designed to be able to support literally millions of concurrent users continuously for very long periods of time. This means:

It can't be prohibitively expensive to serve all of those users.

An error or performance bottleneck while responding to one user shouldn't crash or slow down the system while it responds to other users.

The system should be able to easily scale horizontally, and it should be able to survive hardware failures.

You should be able to inspect and alter the system while it's running, without stopping the entire system.

Object-oriented programming languages tend to use the traditional locks-and-threads concurrency model. In the context of a web framework this could manifest itself in a few different ways.

To take one possible setup, maybe you spawn one OS process per core, and each of those processes spins up some fixed number of threads, which make up your thread pool. You handle requests by taking the next available thread from the pool, and returning the thread to the pool when the request has been processed. You don’t want to spin up an arbitrary number of threads, one for each new incoming request, because this opens your server up to Denial of Service attacks. An attacker could hammer your server with requests until you’ve spun up enough threads to consume the entirety of your machine’s memory.

In this model if all of the code in your web framework is threadsafe (as it is in Rails) and all of your application code is threadsafe, you should be fine as far as race conditions, lost updates, etc.

Note that in Ruby-land that you can't actually execute more than one thread concurrently unless you're running JRuby or Rubinius, and that regardless of language it is a non-trivial use of resources to spin up OS threads and/or processes.

Over in Elixir/Erlang-land things look a lot different. Among Erlang's basic data types, right next to integers, floats, strings, and whatnot, is the Process. Not an OS process, but one native to the Erlang VM. These BEAM processes are super lightweight, with a memory footprint of only a few kilobytes, and they take almost no time to spawn.

It is not unusual for a large Erlang system to spawn hundreds of thousands of processes. A modern MacBook can spawn millions of these processes. Why is that cool?

Memory Independence

Each of these processes is memory-independent of all others. They communicate like objects in an OOP language: by sending each other messages with arguments. But there is no need for locking via mutexes, because there is no shared, mutable state (processes cannot access variables set in other processes).

This style of concurrency is called the Actor Model, and some people believe it is a sort of idealized form of object-oriented programming. Objects are supposed to encapsulate their state, and NOT leak it out into other contexts. Akin to this interpretation of OOP, actors in the Actor Model should communicate only by sending messages. Erlang's implementation of the Actor Model forces you to program this way. 

To show this working in a contrived way, the code below shows one Elixir process spawn another, store the new process’s pid, and send the message {:ok, “hello”} to it. The new process awaits messages of that {:ok, “hello”} format within its receive block, and responds accordingly.

 defmodule Example do    def listen do      receive do        {:ok, "hello"} -> IO.puts "World"      end      listen    end  end  iex> pid = spawn(Example, :listen, [])  #PID<0.108.0>  iex> send pid, {:ok, "hello"}  World  #{:ok, "hello"}  iex> send pid, :ok  :ok

A key advantage to memory independence is that any process crashing need not affect any other running processes, unless you want to link certain processes to one another. There are all sorts of facilities within the language for doing this: once a process has crashed, Erlang provides a variety of strategies for automatically, intelligently restarting the process (or not) in a good state along with whatever related processes you specify. (Look up Elixir Supervisors if you're interested).

Context Switching

Now that you know about BEAM processes, here is how they are used in practice. Your Elixir/Erlang program will run a single BEAM instance per machine in an OS process. BEAM will automatically spawn an OS thread for each CPU core on your machine. The job of these threads is to schedule work for BEAM processes. This happens by way of lightning-fast context switching, similar to how your OS would schedule work for different threads to handle.

There are two awesome consequences of this setup:

Responsiveness - The schedulers will give each BEAM process a tiny window with which to execute its code before swapping out another process. This means that any process that is performing some long-running task (maybe it's performing some intensive computation, talking to the filesystem or network, or maybe it's stalled because of a bug or error) will not prevent any other process from performing its job.

Bang for Your Buck - You can, with modest hardware, quickly respond to hundreds of thousands of concurrent users. What's App famously used Erlang to serve millions of concurrent users with a small number of engineers and relatively modest hardware.

In a traditional threaded environment the size of your thread pool might be a few dozen or hundred threads, and so serving that number of concurrent users would require scaling horizontally to many more servers, probably at a substantially higher cost.

Speaking of which, should a single Elixir/Erlang node not be enough to handle your level of traffic, scaling horizontally is a breeze with BEAM. BEAM processes don't care which machine other BEAM process are on. The mechanism for communication is still the same - passing messages with arguments.

With a few config file changes you can enable multiple BEAM processes to communicate securely over the network via the same inter-process communication they use on a single node.

Consistently Fast Code

Typically when diagnosing the performance of your web application it's useful to look not only at your mean or median response time, but also at your 95th percentile response times. These very slow response times are terrible user experiences, and often they are caused because that request happened to trigger a run of the garbage collector.

In the Rails community you may have heard of "stop the world" garbage collection, where your entire application slows down because GC needs to clear out a bunch of stuff.

This doesn't happen Elixir applications, where each BEAM process's tiny memory footprint is garbage collected independently of all other processes. These GC runs take almost no time, and don't effect other running processes. The equates to far more consistent server response times.

In Closing

So now when you hear someone (probably me) spout off about how cool Elixir is and how it’s so “scalable” and “high performance” you won’t have to wonder what those vague terms actually mean. You’ll know that it is Erlang’s VM, BEAM, that provides the framework for efficiently handling a large amount of concurrent work.

- Dan

[murphydbuffalo]

Breaking the rules

Recently, before leaving a Node.js codebase for a new job working with Ruby on Rails, I began re-reading Sandi Metz’s Practical Object-Oriented Design in Ruby to get a refresher on Ruby and Object-Oriented Programming. This was one of the first programming books I had ever read, and one I wanted to revisit both because it was a favorite of mine and because I remember not fully grasping much of what was discussed in my first time through the book.

For example, the last chapter in the book is on unit testing, and I remember thinking “Testing? ¿Porqué? I can just run my script and check the output, or open up my browser and see that the one or two pages look normal.”

I had no concept of how complex large applications could be, or how valuable of a thought and design exercise writing tests could be. Given that I didn’t understand really anything about testing it isn’t surprising that when the chapter began discussing when to use mocks and stubs, and decoupling your object under test from all other objects, my eyes glazed over and I didn’t really take away much from reading that portion of the book.

Re-reading POODIR (which I love using as the abbreviation for this book because I imagine it is pronounced “Poo-der” and I am an infantile human being) as a more experienced developer lead to many moments where I would take away something I had misunderstood or under-appreciated when I was reading as a beginner.

And by far the most striking of those moments came from the chapter discussing when and why to create abstractions in your code. “Abstraction” in this case simply meaning a general way to handle similar problems. One way abstraction manifests itself is through the oft-repeated credo Don’t Repeat Yourself: if you have highly similar functionality in multiple places in your code, make some reusable functions, classes, or modules, and eliminate that repetition. 

This is a great rule to live by most of the time. It allows you to minimize the number of places where things can break, and the number of times you will need to make changes. It allows you to compose new functionality from different abstract pieces.

What was so interesting to me in this second time through the chapter was not the old and familiar rule of DRY, but that I was only now (roughly two years after my initial read-through) giving equal weight to the other rule Metz discusses regarding abstraction: the Rule of Three.

This is the idea that you should wait for at least three concrete instances of a problem and solution before you attempt to create an abstract solution. This rule says that you should wait until you have as much information as possible before going forward and implementing your abstraction. 

I clearly remember reading this second rule for the first time, nodding my head and saying “that sounds reasonable”, and the quickly forgetting all about it. In contrast, I took DRY to heart and applied it dogmatically in all of my future work. Why the difference in enthusiasm for these two pieces of advice? Looking back, the answer is that I lacked the work experience to appreciate the more nuanced usage and benefits of the Rule of Three.

DRY provides very clear guidelines for how to write your code, and it allows you to reap clear benefits immediately: the ability to re-use a bit of code. I think is by-and-large this a good thing. Making code easier to understand with a descriptive method or function name, and more reusable via the use of that method or function is usually great. 

The Rule of Three, on the other hand, takes into account more than just the perspective of the programmer. It considers the needs of the business or organization that the programmer is trying to help. The Rule of Three wants you to ask if the effort and time needed to conceive of, implement, and refine your abstraction actually justified by the problem you face. It tries to save you, and the person paying you, from writing the abstraction and then needing to re-write it when you encounter the next corner case or level of nuance.

Where the rule does concern the programmer, it offers a less tangible reward: not having implemented a bad or unnecessary abstraction. It’s hard to prove a negative, and the intention “Look! There is duplicated code here and I’m intentionally not doing any thing about it, I’m being patient and avoiding implementing the wrong abstraction in order to make the best possible use of my time.” isn’t always self-evident in your code, particularly if a less experienced developer is looking it over.

They say that beginners need hard-and-fast rules while experts need the freedom to make judgement calls, bending or breaking rules as called for by the current situation. This nuanced approach, of balancing the desire to keep your code DRY with the benefits of delaying your abstract designs as long as possible is certainly more in the realm of the expert than the beginner. I believe that is why it was so much harder for me to embrace the Rule of Three than Don’t Repeat Yourself.

A good example of all of this is when I started at the new job using Ruby on Rails that I mentioned at the outset of this post. I was so eager to prove myself that I rushed into creating an abstraction where the time spent wasn’t truly justified. We had two classes that shared some behavior, and upon seeing this I immediately went to create a new class that would handle this type of situation. It worked, but it took a not-insignificant amount of time to create, including time from one of our senior developers to provide review and feedback.

Trying to prove my competency as a new member of the team I wanted to point to my sparkly, new metaprogramming-filled class and say “Look! I know some cool tricks, don’t you think I’m smart?!  LOVE ME, PLEASE LOVE ME.” .

As it turns out, months later there hasn’t been another case that’s arisen where this new abstraction can be applied. All of that extra time and effort spent, and it doesn’t appear to have been worthwhile. 

As with all things in software, you will know the most about your problem space the further along you are in your work. Said another way, as more concrete instances of your problem arise and get solved, the better you understand those types of problems, their corner cases, and the patterns needed to solve them.

So, go forth and judiciously break the rules! Intentionally leave things un-DRY when appropriate, and forgive a novice the next time they are eager to abstract everything to hell the at the first sign of repetition.

Dan

[murphydbuffalo]

It's like Uber for unprofitability!

Alison Griswold recently wrote an interesting story on Quartz delving into some of the hang-ups with Instacart, a grocery delivery service that pitched itself to investors as the “Uber for groceries”. As Griswold points out in her article, Instacart is one of many “Uber for X” companies that claim to apply Uber’s business model to another industry.

However, just because Uber has found a model works for ride-haling doesn’t mean other businesses can successfully apply that model to other sectors.

Griswold’s article details many of the strong points of Uber’s business model, such as network effects, crowdsourced labor, relatively high volume of sales (detailed below), and profitable margins (they make money on each ride).

Network Benefits - Advantages that come with growing in size. The more drivers and users of Uber, the greater likelihood that a driver is near a user who needs a ride at any given moment. That means less time spent without a passenger for drivers, and less time spent waiting for users.

Crowdsourced Labor - The ability to quickly enlist drivers as contractors who provide their own vehicles. Treating their workers as contractors allows Uber to avoid many of the traditional costs of having employees such as paying into Medicare and unemployment insurance.

While there is a raging courtroom debate over whether Uber’s drivers are employees or contractors, Uber argues for the latter by saying that they provide little to no training to their workers.

And of course not having to purchase, fuel, and maintain cars is a gigantic cost savings.

High Volume - In the context of comparing Uber to something like Instacart, people request rides pretty frequently when compared with something like ordering groceries.

If right now you are saying, “They make money on each ride… no shit! They are a business, who in their right mind would start a for-profit business that doesn’t make money!”, then it’s safe to say we’re on the same page.

Other companies who want to be “Uber-for-X” (there was Prim for laundry, Handy for home cleaning and more), need to look hard at the viability of their business model.

What are they competing on? Will they be more affordable than the competition? Will their product or service be of higher quality than an existing competitor, or will it offer something entirely new? Will it be more convenient to use?  

Uber is arguably an improvement in all three areas, price, quality, and convenience, over traditional car services. In other words, Uber is competing with an old industry that got too comfortable with having few competitors (depending on where you are) or few competitors who were doing anything differently. 

Are other “Uber-for-X” companies in industries that are as ripe for change? Are laundromats and grocery stores bloated monopolies with too-high prices, poor service, and low-quality? I would say no, particularly in the case of grocery stores, which have famously tiny profit margins.

This seems to be another case of the blunder of chasing user acquisition and VC funding without a clear end game. In the case of Instabasket who is now increasing rates after rumors being unprofitable (and relying instead on being subsidized by VC funding), wouldn’t it have been better to aim for profitability first?

Yes, by relying on VC funding as a subsidy you can try to price competitors out of the market, and then shoot for profitability once they’re gone. But if there never was a worthwhile business model to begin with, what have you accomplished?

What this story really put into my head is the notion that if you’re going start a company, if you’re going to take a huge risk, and pour every ounce of your effort, thought, and creativity into some venture …don’t you want it to mean something? If you aren’t a charitable organization, and you aren’t providing something valuable enough that people want to pay you for your efforts, then what’s the point?

[murphydbuffalo]

Do you work with other humans? Read this.

Charles Duhigg wrote an amazing piece for the New York Times a couple weeks ago that details the how and why of working effectively with other people. 

The story follows a woman, Julia Rozovsky, from her time at Yale’s MBA program to a special Google initiative called Project Aristotle. Project Aristotle’s purpose was to figure out why some teams within Google were extraordinarily productive, while others were less so. A real sticking point early on in Project Aristotle’s research was that there were no obvious patterns to be found regarding what made a team effective or ineffective. Should you mix personality types within a team, or group like-minded people together? Does there need to be a clear leader to head the group, or is rule-by-committee fine? Is it as simple as finding the smartest people imaginable and sticking them in a room together? Selecting teams at Google based on any of those criteria would not have pointed you to the most effective teams.

I found this article particularly compelling because it gives you both practical actions you can take at work to improve the happiness and productivity of your team, and the research that backs up the efficacy of those steps.

Here are the points I found most interesting in short form, all building up to the key piece of wisdom from the article in the final bullet point below:

The “collective intelligence” of a given group is not simply the sum of the IQs of the individuals in the group. You could (and Google did) have a team made up of super stars that would underperform a team composed of worse individual performers.

Group norms, the established ways in which a group encourages or discourages certain behaviors, typically override any one person’s individual tendencies. An outspoken extrovert may not dominate a conversation as they normally would in a group that frowns upon that behavior.

Conventional wisdom around grouping people with similar personalities doesn’t hold water: introverts and extroverts can be mixed with no ill-effects on group productivity or happiness, so long as the group norms don’t allow the most extroverted or assertive people to drown-out the opinions of the introverted people.

OK, so if it’s not personality type or intelligence, what is the recipe for getting the most out of a group of people? According to Project Aristotle  it is emotional intelligence. Eventually the researchers found that the most effective groups all had norms that encouraged empathy and non-work-related socializing. The best groups would joke around during meetings, share the intimate details of their personal lives, and they genuinely cared about one another and what they were trying to achieve together. These types of teams had created “psychological safety”, an environment where no one is afraid to speak up and no one will prevent someone from doing so. In a psychologically safe environment novel ideas are proposed and tried out. Concerns and potential problems are voiced early on, rather than hidden for fear of sounding stupid or being judged too harshly. Creativity flows, and people feel accountable to one another. 

Sure, some of this sounds obvious when you say it aloud. But really, if this wasn’t confirmed by several years of research by some of the smartest people in the world, it might have sounded a little too warm and fuzzy to be taken seriously, right?

For me personally, it is incredibly encouraging to hear that being kind and fun-loving isn’t something that comes at the cost of being effective ...rather it is something essential to it. 

So, when you’re tackling that next big project consider getting everyone together for a happy hour first. Aside from giving you a chance to enjoy a delicious Red Stripe, it just might be the most important step your team can take towards achieving that goal.

[murphydbuffalo]

This Week I Learned - 12/23/2015

The Christmas edition (and the first edition)!  

Hi friends,

True to my heritage as a Barbershop Labs apprentice I’m going to do a series of weekly posts detailing something I’ve learned in the realms of programming or technology. These posts will be short and sweet, and I’ll generally just reference an article or piece of documentation I’ve come across in the hopes you’ll find it as useful as I have. This week’s TWILs come from the lands of Ruby and Rails. So, without further adieu:

Proc#parameters

Jamis Buck talks about Proc#parameters. TLDR; this is a handy method for viewing the type and name of the arguments for the proc. Type can be :req (required), :optional (an argument with a default value), or :rest (a splat argument that accounts for the rest of the arguments you pass following any required or optional arguments). 

Refinements

Yehuda Katz talks about safer monkey patching. This is a great read that not only describes Ruby’s Module#refine instance method and its uses, but which also discusses the dangers of monkey patching (complete with a real world example, huzzah!). TLDR; monkey patching (re-opening and modifying an existing class. Generally done on-the-fly from within your application code) is dangerous because it can break anything that relies on the code you modify. You are essentially making global changes. Ruby refinements let you scope your monkey patching to a given class or file. Refinements are defined within a module, and they only effect classes that specifically include that file with the special using keyword. In this way refinements allow you to modify a class and rest easy knowing that your changes won’t break things that depend on the modified class.

Hash#slice

A Rails instance method for Hash.  Get a subset of a hash. Pass in some keys and get back a hash consisting of the corresponding key-value pairs for those keys from the original hash.

[murphydbuffalo]

Regular Expressions: Don’t be greedy

The other day I was reading the chapter on regular expressions from Marijn Haverbeke’s gregcellent book, Eloquent JavaScript. Yes, regular expressions get an entire chapter in that book... All hail Lord RegExp.

Until reading this section of Eloquent JS I hadn’t heard of greedy and non-greedy regular expressions, and the difference between the two. Turns out it’s an important distinction that can help you to leverage the capabilities of regular expressions while minimizing their notorious difficulty-of-use when matching complex patterns. That difficulty-of-use has been noted many times, but I like the way Haverbeke phrases it best:

“Regular expressions are a sharp tool with an awkward handle.”

In other words regexps have the potential to be tremendously useful, but when the use case is non-trivial they are often misused or used to poor effect. Making use of non-greedy regular expressions will not change this fact; regular expressions will still not be viable for many complex use cases. But doing so certainly can help to expand the number of practical applications that regexps are appropriate for.

A non-greedy regular expression is...

A regular expression that examines as few characters as possible in order to match each of its patterns, allowing it to immediately search for matches to its next pattern without backtracking. Wut? Backtracking? Such confuse.

When given a string to match, regular expressions’ default behavior (at least in JavaScript, Ruby, and Python, the languages I tested this out on) is to match as much of that string as possible for each pattern specified by the regular expression.

When using quantifiers such as + and *, which match more than one instance of a pattern, the regexp will sometimes be able to match the entire string with that quantifier pattern. If that pattern is not the final pattern in the regexp this means that the regexp needs to backtrack, or examine the characters of the string from last to first in order to find a match for that final pattern.

An example will make all this clearer, but first a quick refresher. Remember that: 

. is the wildcard operator, indicating any character except a newline.

The | operator is a logical OR statement.

[ ] enclosing patterns means “match any of the characters included here”. Prepending a ^ to the characters inside the brackets means “match anything except these characters”.

( ) are used to group patterns together.

You can use flags to set option on your regexps. The g flag is for global matches, meaning find all occurrences of the match in the string. They go after the closing backslash, eg /match this/g.

? is the “optional” operator, indicating that the regexp should attempt to match 0 to 1 of the preceding pattern.

* is a quantifier, meaning that the regexp should match any number occurrences of the preceding pattern (0 to infinity). 

+ is another quantifier, meaning that the regexp should match a minimum of one occurrence of the preceding pattern (1 to infinity). 

And here’s the big one: when ? follows either the * or + quantifiers that quantifier becomes non-greedy. Eg, \w*?.

With that out of the way let’s say that you have the regexp 

/(.*)(tasty bagels)/

and you attempt to match the string 

“This is a lovely sentence that ends with tasty bagels”

Here are example in Ruby, Python, and JavaScript.

The regular expression engine will by default attempt to match as much of that string as possible with the (.*) pattern of the regular expression before attempting to find a match for the lovely bagels pattern. In this case the entire string consists of non-newline characters and so it can be matched in its entirety by that first pattern, (.*). 

Only after reaching the end of the string, while still searching for matches to that first quantifier pattern, does the regular expression engine attempt to satisfy the second pattern, tasty bagels. So, it backtracks one character at a time until it discovers that, indeed, the end of the string does match that pattern. 

If that smells like an inefficient and illogical mess to you then you’re not alone. The intent of this regular expression is clear, we are trying to say “search this string for ‘tasty bagels’ preceded by an arbitrary number of non-newline characters”. Our intent is not, “find as many arbitrary characters as possible, followed by ‘tasty bagels’.” But this is how greedy regular expressions treat patterns with quantifiers, they aim to match as much of the string as possible with those patterns before moving on to the next pattern.

Bagels, take two

It doesn’t take much to change the example regular expression used above into a non-greedy one, simply append a ? operator to the * quantifier so that it looks like this: /(.*?)/. As noted above, appending the ? operator to any quantifier will make it non-greedy. What does that mean in the context of our example above? Not much. The matches that are returned will be exactly the same, it is only the manner in which our regexp found those matches that has changed.

The regular expression engine will no longer use the first pattern to match the entire string, and then backtrack once the end of the string has been reached. Our non-greedy regexp will match as little as possible to satisfy the (.*?) pattern, and immediately thereafter attempt to match the tasty bagels pattern.

This results in an implementation that isn’t any prettier than our initial attempt. The regexp looks at the first character, sees that it is in fact a non-newline character, and then attempts to match “tasty bagels”. It won’t find a match for that pattern beginning at the second character, so it backtracks and attempts to include that second character in the first pattern. This process repeats over and over again until the first occurrence of “tasty bagels” appears at the end of the string.

While the internal mechanism of this non-greedy approach isn’t any more elegant than the first (aka, there’s still a ton of backtracking), at least our intent is more accurately represented. We want to find “tasty bagels”, possibly preceded by a bunch of alphanumeric characters and whitespace, and that is now roughly how the regexp goes about things.

Intent is important

Alright guy, explain why I should care.  Wellp: 

In the previous example we got the same matches regardless of whether we were using greedy or non-greedy regular expressions. But that is not always the case. In fact, in scenarios where the pattern you want to match appears multiple times in the string, it is very frequently the case that you’ll want to use non-greedy expressions to achieve the proper matches.

Let’s go by way of example: say you have a string representation of some HTML elements and you want to manipulate the contents of every <p> tag in some way.

After a moment spent pondering you might write a regular expression like this:

 /<p>.*<\/p>/g 

reasoning “I want to match every instance of an opening <p> tag, any number of characters in between, and then a closing </p> tag”.  But slow down, cowboy! Y’aint quite there yet***.

First of all, what about multi-line content between the paragraph tags? The . character only matches non-newline characters, so what you really need to say use is the m flag in Ruby or Python, or the super-not-friendly-looking [^] pattern in JavaScript. The m flag says “make the . operator match any character including newlines”. 

What on Earth is that [^] mess, you say? It translates to “match anything that is not in an empty set of characters”, aka everything including newlines. And that is JavaScript’s only means of replicating the functionality of the m flag because JavaScript is silly.

Ok, so we’ve updated our regexp to the following in Ruby or Python: 

/<p>.*<\/p>/gm 

or to 

/<p>[^]*<\/p>/g  

in JS. If we attempt to match the string 

<p>I'm a <em>paragraph</em>.</p> But I'm not. <p>I'm one too!</p>

we’d hope to find that all <p> tags and their contents are matched, and they are. But look at what the matches returned! In Ruby, Python, and JavaScript the result is the same, we’ve matched more than we intended. The “But I’m not” part of the string is not enclosed in <p> tags, so why did our regexp match it?

As you might have suspected, our expression matches the first opening paragraph tag, then the greedy .* statement matches all characters up to the final closing paragraph tag, including the first closing paragraph tag and the second opening paragraph tag. Shit! Shoot. Dangit. No problem though, we now know the regular expression tool that can help us avoid such over-eager pattern matching.

By instead using a non-greedy .*? statement as our quantifier we can tell the regexp engine to match closing paragraph tags as soon as possible, resulting in two separate matches for paragraph tags and no matches for content outside of those tags. Huzzah! Here are working non-greedy examples for Ruby, Python and JS, meaning that you can now match multiple instances of patterns with a specific beginning and ending sub-pattern. 

Apart from matching multiple sets of HTML or XML tags this solution can also help you to match multiple groups of multiline comments in your code, or really any other content that appears multiple times and always begins and ends with the same pattern. Perhaps certain articles or comments with a common header and footer fall into this category.

Now before you get all hopped up on regular expression juice remember Haverbeke’s warning:

“Regular expressions are a sharp tool with an awkward handle.”

And in this case that awkward handle takes the form of nested tags/comment blocks/etc. You’ll notice that in the provided links to example patterns matched with these non-greedy expressions that nested tags do not match properly. As you can probably guess by now, those nested tags are problematic because the first closing tag encountered will always satisfy a non-greedy regexp, and prevents us from finding the final closing tag in the nested set of tags.

Maybe you’ve encountered this Stack Overflow answer that gets into the silliness of trying to do too much with regexps, in this case trying to parse HTML properly.

So, you’ve been empowered and you’ve been cautioned :) That’s all! Happy coding.

Dan

***yeah, I totally tried that one at first, don’t feel bad ...or maybe feel good because you didn’t.

[murphydbuffalo]

git, I love you.

Hello dear reader, so good to see you again. I had heard you were pining for some sweet, sweet git commands to satisfy your need for version control greatness. To that end, I have a riveting story just for you.

About once a week I’ll look up from my computer (period, end of story, it’s a sad existence) and crane my neck over towards my fellow git-loving coworker, Richard. The conversations typically go something like this:

“Ricky!  Man do I wish git had a way to do X, then see Y lines of code in Z format. That would be ducking awesome.”

“Oh, have you tried git xyz?”

“Ohmehgard, git xyz is the BEST.”

Yeah. We have fun.

As a celebration of those wild dialogues, I’ve listed below 5 of my favorite git commands for your programming pleasure. Calm yourself, and take a deep breath before continuing.

Style guide woes

Recently, my company dug itself out of the hole that is writing your own JavaScript style guide. That misadventure began nobly enough, with the team wanting to embrace unfamiliar style conventions not found in most popular style guides in the name of improving code readability. For example we began enforcing more liberal use of whitespace inside of parentheses ( like this ). 

That misadventure ended with a period of experimentation followed by continued discussion about which parts of the style guide to keep, which to abandon, and the realization that we were needlessly debating something has been handled well many times before by those aforementioned mainstream style guides.

In the end we agreed upon the lovely airbnb style guide, which serves us quite well, and keeps our code in line with established style conventions. 

The transition from our home-grown style guide to airbnb was made significantly less painful by the awesome JSCS library, which is a code style linter and formatter capable of fixing many style irregularities programmatically. You simply specify the style guide you’re using and any rules you want to ignore or add in and JSCS goes to work.

Enter git add --patch

For the style guide violations that JSCS couldn’t handle on its own, manual fixes were required. The horror! As a faithful abider by git best practices I know that The Correct And Virtuous Path of Version Control Goodness (aka the CVPVCG) tells us to keep our commits atomic, or concerned with only a single topic, in order to allow for quick identification of problem commits when debugging or doing code review.

However, I am also a chimpanzee masquerading as a human and I found myself correcting style guide violations in various files as I was working on changes unrelated to style. Yikes!  What to do now? I have unrelated changes in the same file (files in this case), but want to keep my commits atomic.

git add --patch to the rescue!  You’re familiar with our old friend git add, no doubt, and I’m sure you’ve used the -A or --all flags to stage multiple files at once. In this situation, adding the --patch flag to git add -A was just what the doctor ordered for filtering out those style changes into a separate commit, before committing the unrelated changes.

The --patch flag will have git break your changes down into what it thinks are reasonable chunks of related code (it doesn’t always get this part right, but it’s pretty damn close) and ask you if you want to include that chunk in the commit. Behold:

This made it easy to go through multiple files, adding style changes and ignoring other changes, allowing me to create one commit with style fixes before moving on to more substantive changes.

git diff --color-words

No long-winded tale of style guide woe here, just good old fashioned reading of the manual by another coworker of mine. 

I don’t use the github Mac GUI because I’m a masochist. Yet I still feel I have the right to complain endlessly about the way in which git formatted the output of git diff. git diff by default will show the diff for parts of file stacked on top of each other vertically. When an untracked file contains many changes I find that the further I need to move down in the page, the harder it is to focus on the changes themselves as opposed to trying to make out which parts of the file have been changed. Here’s the default output from git diff:

Adding the --color-words flag helps alleviate this admittedly minor problem by showing the diff for a line in-line with the original code like so:

Of course this is nothing more than personal preference, and most sane people who are concerned with the readability of their git output use a GUI. So, --color-words is probably the least useful of any command or flag mentioned in this post, but it does help to keep your diff as spatially small as possible.

git stash save [message]

If you’re like me you make frequent use of git stash to temporarily hold on to changes you’ve made, but aren’t ready to stage, in order to switch to another branch or take another approach to solving the problem.

Using git stash list will give you a list of all your stashed changes, identified by number, the branch the changes were made on, and the title of the most recent commit message made prior to calling stash. It looks like this if you’re not familiar:

If we’re being honest, reading through a numbered list identified by branch and commit message title isn’t a great way to identify the change you’re looking for. What if you have multiple stashes on the same branch and commit as in the example above? A better way to stash your changes is to provide a descriptive message, just as you would with a commit. This can be done with git stash save [message]. This gives us output that is much easier to navigate:

git blame [filename]

The name says it all really, because we are vile, base creatures, and sometimes all we want to do is say “Who on Earth wrote THAT line?!”. Well, use git blame [filename] (and and the -c flag if you want more readable output) to see the commit hash and author for every line of a given file. Boom, roasted:

git log [filename]

This may be an obvious one, but I for one had been using git log for a very long time before Googling “see all commits for a file” and smacking my head upon seeing the obvious solution. git log is great for seeing a history of what’s been done, hopefully accompanied by short-and-sweet commit message titles, and very descriptive commit message bodies.

Being able to do this for a single file fulfills the same function as git blame in many ways, showing you how different people have shaped a file over time. Beyond that it is also great for helping you focus on a small subset of changes when using log as a means for understanding the path of development or when rooting out a nasty bug. Take a peak:

Farewell, sweet prince

That’s all for now folks, thanks for stopping by!

[murphydbuffalo]

Meta World Peace (Metaprogramming!)

Welcome dear reader!  Today we tackle metaprogramming in Ruby, what it is, why it's useful in your everyday programming life, and what the basic concepts, techniques and resources available to you are.  

In brief, metaprogramming is writing code that will generate additional code as your program runs based on what external data is provided.  Ruby provides various tools allowing you to examine and alter existing modules, classes and methods or to add brand new functionality programmatically, rather than by hand.  

So, why should you learn this stuff?  Well one of the primary uses of metaprogramming is to DRY out your code, as it is excellent for eliminating the repetition of highly similar, but not identical, functionality from your codebase.  This process typically involves some logic that is repeated throughout your program with slight variations each time it is called.  Relying on user input, a database schema (think of how ActiveRecord defines methods for you that provide access a parent model's child objets, or that allow you to search by specific properties of a model), or other external data source, you can automatically have your code define unique variations of these chunks of logic during runtime.

DRY'ing out your code is well and good, and is an important skill for any developer.  But a more compelling reason for familiarizing yourself with metaprogramming is to increase your Ruby literacy.  I believe this is one of the most effective ways of improving your understanding of other peoples' Ruby code, be it Rails or one of any number of popular Gems.  When debugging these third-party libraries you'll find that metaprogramming is commonplace, and therefore understanding it is a major part of understanding what is actually happening in the code.

Below are some examples of essential metaprogramming techniques, with images courtesy of the wonderful RubyMonk.com.  Ruby Monk has an excellent guide on this topic which I can't recommend highly enough.  Code School's "Ruby Bits 2" course also has great material on the subject that I found extremely helpful.

1) Code inspection, send and method_missing:

Oftentimes, in order to be able to generate code on-the-fly you'll need to perform some checks on the existing code you're dealing with.  For example, what methods does a certain object respond to, what self currently is (the scope or context), or what variables are currently defined within a given scope.  This is code inspection.  

These tools are very handy, and while they don't dynamically generate any code for you, they are invaluable in situations where you want to do so, because they allow your program to examine any piece of code, without knowing its contents ahead of time, and determine what course of action to take when adding or modifying functionality on-the-fly.

Fairly self-explanatory are Object#methods and Object#public_methods, which return arrays of all an object's methods or public methods, including those it inherits from parent classes.  If you pass false to public_methods it will return the object's public methods, but none of its ancestors'.  You can also call Object.method(:some_method_name_goes_here) to return an instance of class Method, which allows you to examine the properties of the method, for example by calling Method#parameters.

If you've used the Gem pry then you've already seen another very useful code inspection technique:  Binding objects.  Kernel#binding (you can simply call binding nakedly, without the reference to Kernel), creates an instance of the Binding class which captures, among other things, the scope and state of the program on whatever line you place it.  You can define getter methods around bindings and place them at various points in your code to reference multiple contexts in this way.

Ruby also gives us some quick ways to access basic info about the program itself.  The __FILE__ method returns the current file name, __LINE__ returns the current line of the program, and the global variable $0 returns the name of the program that was initially executed to arrive at that point in the code.

Get giddy, beloved reader, because you're about to add two of the most useful methods in Ruby to your toolkit: send and method_missing.  Sticking with the theme of being able to deal with code without knowing its exact nature in advance, send is your friend, it allows you to call any method, passed in as a string or symbol, along with any arguments or block it may require.  

This functionality is great for delegating methods to other objects when a particular object doesn't know how to respond or if you simply want to extend another class's functionality.  But how does this work exactly?  As you may know, when you call a method in Ruby a search is made through the method lookup chain (check out the image below) for that method.  First the class of the receiving object is searched, followed by any modules included in that class.  If the method isn't found the search is repeated in that class's parent class and its modules, with this process repeating all the way up Ruby's class hierarchy.     

The Ruby method look up chain. From the allenlsy blog <3

However, Ruby also checks within each class to see if method_missing is defined.  If it is, the logic in that method will be executed in lieu of the original method that was called.  This is amazingly helpful as you can define custom behavior for when an unknown method is called on one of your objects.  Inside of a particular class's method_missing you'll often see calls to other classes via send, effectively saying "if my class doesn't know how to deal with this particular method, send it to another class I know can handle it".  This might seem to be a strange use case, but imagine it in the context of metaprogramming.  You will have methods generated dynamically based on specific conditions in your codebase, and it will not always be clear exactly what methods are defined for a given class.  

2) eval is evil

To get the value of some arbitrary piece of code you can pass the eval method a string and a Binding object to specify the scope (eg, is this a variable from the global scope, or the one with the same name that you've defined within a method?), and it will fire up the interpreter and parse that string as code.  This means that, as with some of the code inspection techniques mentioned above, you don't need to know ahead of time what code will be passed in in order to evaluate it... pretty sweet!  

But when you encounter eval in your meta-studies more than likely the only thing you'll hear is how you should never use it and how eval is evil.  So why all the hate?  Take heed!  eval is computationally very expensive because you need to fire up the interpreter, and it's also highly insecure, as you're allowing someone to pass in potentially malicious code to be blindly executed.  

So, what's a dev to do in this situation, where you need to examine a dynamically generated bit of code, but don't want to expose yourself to the pitfalls of eval?  Best practice here is to pass a string to const_get (called on a class or module) or instance_variable_get (called on a specific instance) to return the actual Ruby object of whatever class, module or instance variable you're looking for.

3) Lifecycle callbacks, instance_eval and class_eval: 

Check out the code example above from Ruby Monk's Metaprogramming book, it uses a pattern commonly found in Gems and one that we'll break down for you here.  

Rearing its head again is the concept of context:  what is the scope for a given object?  The instance_eval, class_eval, and module_eval methods give you safer (although still expensive) alternatives to eval, and are great for modifying existing objects.  Where with eval you pass in the context you want to evaluate via a binding, these methods set their context to the object they are called on, and then define some functionality for that context via a passed in string or block of code.  

instance_eval defines methods particular to a specific instance, so class methods when called on a class, and singleton methods when called on instances of a class.  class_eval can only be called on a class or module, and defines instance methods on that class or module.  module_eval is simply an alias for class_eval, and obviously is more readable/semantic for defining instance methods within modules.

You can see that in this example, we're extending the ClassMethods module and including the InstanceMethods module into an object, using instance_eval and class_eval to add those modules within the correct context.  instance_eval, you'll recall defines methods exclusive to a single instance, so that means class methods when dealing with a class object, and conversely, class_eval defines instance methods.

But the real clever part to this code is the use of included.  included is an object life-cycle callback, or life-cycle hook.  These are methods that Ruby will call automatically after certain events occur in your code.  There are callbacks for when a module is included or extended, when a class is inherited from, for when a method is added or removed, and even for less common situations such as when a method that was defined becomes undefined.  So, when you define include as was done in the example, you're saying run this code whenever the Gym module is included in an object, then take that object as a parameter and perform some operations on it.

Putting all of that together, the above code says that when you include the Gym module in a given object, take that object (a class presumably) and  extend the ClassMethods module as a set of class methods (so, in the context of a specific instance of a class), and include the InstanceMethods module as a set of instance method (so, in the context of that instance's class).  This provides a succinct way for third-party library authors to add both class methods and instance methods into a class with only one call to include.

4)  define_method and instance_variable_set:

Finally, we have this scrumtrulescent beauty of a code snippet.  Here, define_method is called, passing in the name of the method to be defined:  initialize.  In this case the method name is a hard coded value, but as mentioned above, names are often passed in from some dynamically generated data source.  You can pass define_method a block, where you have access to any arguments that will be passed to the method and can define the method body.

In this Ruby Monk example we're creating a module that can be included in any class to provide a hash initializer (a common design pattern allowing you to supply arguments to initialize in any number and order).  This allows us to write our initializer in only one place, as opposed to in each class, DRY'ing out our code.  Let's see how they go about doing it.

First, the hash_initialize method takes a splat argument, allowing you to pass in as many arguments as needed.  That array is then compared to the array keys for the hash that will be passed in to the dynamically defined initialize method (that hash is accessed as the variable h within the block passed to define_method).  This is done to prevent initializing instance variables that have not been specified in the call to hash_initialize.  Now if any arguments are not passed to hash_initialize, but are passed to initialize, the missing array will contain those extra arguments and an error will be thrown.

If no extra arguments are passed to initialize then each of the key-value pairs of the hash passed in to initialize will be made into instance variables for the object being initialized.  This is done via instance_variable_set, which as the name implies, accepts a variable name and value as arguments.  Note the use of string interpolation here to add the "@" to our instance variable name.

The end result is very terse, but allows every class to have a hash initializer, with control over what instance variables are allowed to be defined, in only two lines of code:  including the module, and calling hash_initialize.

WHEW.  Did you make it all the way through?  Kudos to you, intrepid reader, if you did.  Hopefully this has been a good high-level intro into metaprogramming, and one that has you all set to dive deeper into this powerful part of the Ruby language.

More from On and On coming ....soon!

[murphydbuffalo]

JavaScript Closures

I was reading this awesome article by James Holmes on delegated events in pure JavaScript (without JQuery), and discovered that I sorely needed a refresher on closures.  

What is a closure?  In JavaScript, when one function contains another function it's called a closure.  In a closure the inner function has access to all variables defined in the outer function, but the inner function preserves its own unique state for those variables, independent of any reassignment occurring in the outer function.  

Common use cases for this include instantiating an array, string, or a variable which acts as a counter (for example within a for loop) in the outer function, and then appending to or incrementing that object in the inner function.  Crystal clear, right?  A code example from the  article mentioned above provides a much better explanation.

Here we have a list with some buttons:

And here we are looping over said buttons and attaching event handlers to each one so that the clicked button will be given the "active" class.  Note the anonymous function being called within the .addEventListener function:

The author goes on to explain that this simple-seeming function will not in fact do what many people would expect.  Namely, it won't bind all of the buttons with a "click" event handler.  Rather, it will only do so with the last button in the list.  What gives?

Well, it turns out that a crucial property of closures is that, when used within a loop, the inner function will only return the final value calculated.  So, the return value of the function ends up being the result of the final iteration of the loop, in this case the final button.

To properly attach event handlers to all of the buttons while using a closure, you would need to make your function explicitly return a value each time through.

Here the author demonstrates a closure with an explicit return of the inner function, ensuring that each iteration of the loop returns a value:

That's all on closures for now, dear readers.  Thank you for stopping by, and for anyone interested in an in-depth look at event delegation in JavaScript, check out the James Holmes article referenced above.

More from On and On coming ....soon!

[murphydbuffalo]

DRY'ing Out Your Rails App, Part 1.

First things first, a big shout out to both the folks over at Code School and our head honcho here at Barbershop Labs, señor Adam Rubin, for imparting some of their Rails wisdom to me.  And without further adieu...

An exclusive peek inside of the Barbershop Labs offices.

The basics:

A couple of mantras every Rails programmer hears chanted from the very earliest stages of his or her career are to keep your code DRY and Convention Over Configuration.  

The first is Don't Repeat Yourself for ye uninitiated, and at its core it is about keeping your code encapsulated in reusable methods/functions, classes and modules.  Identical or highly similar functionality then only need to be edited in one place in order to cascade changes throughout your entire application.  Furthermore, giving semantic names to these snippets of code helps to keep your code readable for others.

The second, Convention Over Configuration, refers to architecting your application so as to follow established file structures, design patterns, and code formatting in order to, much like DRY, make your codebase easy for others to understand, debug and expand upon.

The Rails Way:

As you become more well-versed in Rails it becomes apparent that there is a lovely union between these two concepts.  Rails encourages you to keep your code DRY in very specific (read: conventional) ways, expanding on the framework-agnostic approaches of extracting code into methods, classes and modules, and providing specific guidance on how to keep an MVC application maintainable and extensible.  Beware!  What follows is certainly not an exhaustive guide on what is considered best practice in Rails.  But it is your humble author's hope that this discussion on single-responsibility models and controllers, as well as the topics covered in DRY'ing Out Your Rails App Part 2, can serve as a good primer for those looking to get a little cozier with "the Rails way" of doing things.

Skinny Controllers and Skinny Models, Too:

A center for ants?

Ensuring that your controllers and models (which are simply classes) follow the single responsibility principle, that all classes should have a single coherent purpose and only a single reason to change, is an important step towards keeping your code modular and easily understood.  Instead of striving for "skinny controllers and fat models" your approach should be to make both models and controllers lean, mean adherents to SRP.

For controllers, this means restricting your methods to the seven standard Rails CRUD actions, index, show, new, create, edit, update and destroy, in all but extraordinary circumstances.  Most any Rails developer will tell you that the responsibility of the controller is essentially to dispatch incoming requests to the appropriate views, referencing the appropriate data and business logic from models.  But many a Rails developer (as I guiltily look in the mirror) have strayed from that path by adding actions which belong either as business logic in a model, or in an entirely new controller.  

For example, complex or unusual functionality with your registration process may require both a Users and Registrations controller.  A controller that makes frequent use of non-business logic conditionals or that makes use of non-CRUD actions could likely stand to be split out into two separate controllers, each with a clear, single responsibility.

The more common scenario requiring cleanup in a controller is extracting business logic into your models (frequent use of business logic conditional statements in the controller is typically an indicator that this needs to be done) in the form of instance methods (sometimes called guard clauses in this context) or callbacks such as before_save, after_save, after_destroy, etc.  But the process of ensuring that your controllers and models obey SRP is not always as simple as shifting code from one to the other.

As with controllers, it is best practice to separate functionality out into two models if that creates a clearer, more modular purpose for that class.  The previous example of a complex user registration process could again apply to your models.  So, rather than one large model, full of logic for altering the state of your Users, you might have two.  One that deals specifically with registration, and another for the remaining logic.  

Continuing with the registration example, the registration model does not need to inherit from ActiveRecord as it can act simply as the User model's interface for logic pertaining to registration by taking an instance of the ActiveRecord model as an argument (example below).  This makes the Registration class significantly more lightweight (because ActiveRecord is, in technical terms, a Bad Mamma Jamma), and creates even more clearly defined encapsulation:  one model encompasses a specific set of business logic, while another interfaces with the database via ActiveRecord.

Well friends, that's all for today.  Coming soon:  Part 2, where we'll discuss Concerns, Scopes and Decorators.  Thanks for reading!

More from On and On coming ....soon!

[murphydbuffalo]

A Lannister always pays his debts* ...and a good engineer should too.

Last week was my first as a junior software developer at Barbershop Labs.  I can't speak highly enough of the experience so far, everyone has been welcoming, fun to work with (seriously!), and eager to help me better my understanding of software development.  

Improving as a programmer necessitates learning by doing, I don't think any reasonably experienced developer doubts that.  However, when I haven't had my head in my text editor, simply being around the senior devs at Barbershop has been an awesome way to learn more about general web development concepts not related to any specific piece of code.  I've been fortunate to hear a lot about the business side and workflow of a consultancy like Barbershop, and the importance of good management for engineering teams.  I'll touch on many of these topics in the weeks to come, starting today some thoughts on technical debt.

Prior to coming on board at Barbershop when I heard "technical debt", I immediately thought of test coverage.  My understanding of the concept was that in a crunch you might need to skimp on writing tests in order to deliver a project or a feature on time - blasphemy, I know!  Your technical debt would then be the tests needed to provide solid coverage for the app, thereby ensuring your ability to add features and refactor without wondering if all of the app is still functioning.  Not setting aside time to write those tests allows you to deploy new features now, making your clients happy.  Unfortunately, this will also ensure that your app is at risk of breaking without your knowledge whenever you implement a change.

But technical debt goes well beyond testing.  Most any tradeoff you make between engineering productivity now, and code maintainability or improved engineering performance in the future is a discussion of technical debt.  Hastily designed or written programs may need significant refactoring to become easily maintainable, expandable and scalable.  Not bothering to schedule time to address those concerns has the potential to derail the project later on in time.  

Developers also need to stay current on the newest technologies, whether that means entirely new, or new versions of, languages, frameworks and third-party libraries.  Refusing to set aside time to do so means that at some point your team will be unable to work as quickly and efficiently as competitors that are using the latest and greatest in their stack.  This not only guarantees that you won't be as productive as you could be, but your employees may well be tempted to leave for greener pastures if there is no apparent effort to pay off technical debt and stay close to the cutting edge.  

Don't be that guy

When discussing these tradeoffs it's easy to imagine the potential for the desires and incentives of managers and developers to conflict.  For the short-sighted manager at a busy firm, when will it ever be a good time to devote significant man hours to learning a new technology, refactoring brittle code, or writing tests for an app that's already been deployed?  Probably never.  For the manager who knows the importance of investing in future productivity periodically, the answer is clear:  there should be regularly scheduled time for paying off technical debt.  The company that has an explicit policy for doing so will guarantee its competitiveness in the future, and the happiness of its employees now.

That's all for this week, thanks for reading!

More from On and On coming ....soon!

*obligatory Game of Thrones reference

[murphydbuffalo]

Why Mission Control Works

Dear reader! It's been too long.  How are you?  Hungry?  For burritos?  Me too.  Chipotle beckons.  But first, let's catch up.

The pace of life inside Mission Control, Launch Academy's lovely base of operations, has changed noticeably as of late.  I've fallen in love with JQuery and its ability to create that smooth, modern experience that users have come to expect from the web.  Rails group projects are getting more elaborate, with the incorporation of ever more features (Gems-du-jour include Carrierwave for file uploads, and authentication with Devise or Oauth).

It seems everyone is conscious of how little time we have left together, how much we still want to learn, and what cool things we might yet build before the end of the program.  Of course, once our cohort's time at Launch is up, none of us is likely to cut ties with former classmates, or decide there is nothing left to learn, or suddenly lose their love of programming.  But something will be different.

Mission Control is a special environment, one that is unusually good at spurring progress.  This is not some mystical quality I'm applying out of self-interest or fantasy, and it is not purely a function of the sheer workload of the curriculum.  Sure, Launchers put in a ton of hours.  Around 60 per week is a conservative average for the cohort.  I've seen many put in weeks of 70 hours and more, depending on their openness to over-caffeination and how obsessed they are with creating cool video games that look like they're from 1992.  

Most people can learn a good deal about nearly anything if they simply practice.  Put in the time, the reps, the effort and you're a long way towards any goal.  So, what's this magic I'm hinting at?  In my mind it comes down to two things:  

1)  Jam a bunch of diverse, inquisitive, friendly people into one place.  Now, make them solve problems together every day for three months.

The exchange of alternate solutions to, and perspective on, problems here is phenomenal.  I love coding.  But some of my favorite moments at Launch have come away from the keyboard.  There isn't a member of the cohort who doesn't have both interesting past experiences, and insights into our current work to bring to the table.  Prior careers include computer networking, medical research, marketing, sales, education, law, the military, and classical music.  When you get up to take a break from the screen, or sit down for lunch, more often then not you continue hashing out the problems of the day.  You switch pair programming and group project partners frequently.  You share what worked and what didn't every morning in Stand Up or at Lightning Talks.  Never before in school or in the office have I been exposed so consistently to practical, actionable and focused feedback from such amazing peers.  

And:

2) Build constantly!

While prepping for Launch you learn a lot, you are supplied with skill level appropriate books, videos and exercises.  You read, and watch and code whenever you can.  But it wasn't until I began working on applications every day that I really knew the best way to learn programming.  Watching an endless stream of Treehouse and Codeschool videos was great.  Those are quality services with excellent content and presentation.  However any developer will tell you that there is no substitute for physically working through some key processes without step by step guidance from a video fresh in your memory.  

It is absolutely essential for a budding developer to learn how to decipher error messages and backtraces.  It must become second nature to reference the correct documentation for your problem.  To know when and how (seriously, sometimes the hardest part is knowing what to ask, and what terminology to use) to look for answers on StackOverflow and the like is an invaluable ability.  Professional programmers wholly embrace version control, adherence to style conventions, writing comprehensive test suites, and properly thinking out user stories.  These are skills that you will be hard pressed to understand at a fine level without diving in and building applications.  

The instructors are there to rescue you from disaster, to provide expert feedback, and to introduce challenges of the highest possible value to you in terms of on-the-job relevance and learning opportunity.  However, the beauty of Launch Academy is that the instructors are not there to guide you every step of the way.  They will not answer every question outright, and you are expected to think critically, do your research, and convene with your peers before expecting a solution to be delivered on a silver platter.

As I explore the Ballmer Curve with my fellow Launchers this weekend I'll be sure to enjoy the company and savor our remaining time at Mission Control.  So far it's been an experience I can't speak highly enough of.

More from On and On coming ....soon!

[murphydbuffalo]

Beyond Square One

One of the first guest speakers we received at Launch this summer cautioned something to the effect of "And no one will be impressed by your first simple CRUD apps, despite all of the cool content you include in them."  That sentiment seemed sensible at the time (after all what senior dev would be impressed by some novice's pet project?) and at that point I hadn't yet glimpsed what actually could impress an experienced engineer.

For ye uninitiated, CRUD refers to Create Read Update and Destroy, the basic functions available for database entries.  While most every useful app is a CRUD app in that it applies those functions, simply because you can achieve those essential behaviors does not mean you're a rock star programmer.  

A CRUD app is something that aspiring developers, such as your humble author, first look at with tremendous pride and happiness.  "Look I built this!  It does something."  I vividly remember the feeling of satisfaction from seeing those first few projects display their data and respond to user input, while maybe showing off a slick-looking front end. 

While far from being able to write the next big, world-shaking, piece of software, our cohort is undeniably taking steps towards a professional level of quality at an exciting pace.  Fellow Launchers Dan Kleiman and Craig McGinely are nearing completion of "The Datastroyer", a compound data structure reader that I bet could get them hired today.  Launchers Nick Lee and Brian Cox are building out a fantastic 2-D game using the Gosu library from MIT.  I'll discuss my personal projects at length in future blog posts, suffice it to say our programs' level of applicability to real world problems and end users is growing quickly.  

Many of the concepts we've covered in our pre-work or at while at Launch had been useful only in hypotheticals until recently.  Now, making use of Git is not solely an exercise, it is the best way to collaborate on projects, to revert back to old versions, and to try out significant changes.  By now nearly all Launchers have experienced the utter confusion of peering into a maze of ill-written code, wishing it had been structured more carefully.  As such, we've come to embrace object-oriented design as absolutely critical to keeping our applications manageable as their requirements change, and comprehensible to others with well named, single-responsibility components.

At this stage we are also beginning to replace some hand-written code with libraries governed by "convention over configuration".  For example, sanitizing user input and writing out each of your Model classes were crucial learning experiences.  Now, having an understanding of those processes we are better able to appreciate, and comprehend, Active Record in all of its meta-programming glory.

So, yes, I'll admit it.  I thought my first CRUD apps were awesome; but what lays ahead, a wide range of more sophisticated projects, has me truly eager to continue learning.

More from On and On coming ....soon!

[murphydbuffalo]

Don't fear the params hash! Part 2 : Params and dynamically generated content

Good to see you again, dear reader.  In the previous post On and On gave an overview of the essentials of HTTP as a way to set the stage for this post:  a discussion of Rails' and Sinatra's params hash.  Please read the HTTP post first if you're not familiar with GET and POST requests and their responses.  

This post will offer a simple explanation of what params is and what you can do with it.  The end goal here is to clarify something that had given your adoring author fits as he put together his first dynamically generated websites at Launch. 

Lists are fun!  Let's start with some lists.

Params is:

An object.

A hash.

A collection of key-value pairs.

Created automatically by Rails/Sinatra after every GET or POST request.

Holds values specified during those requests.

Params is not:

Voodoo magic.

Something aspiring web developers don't need to understand.

Something that is not very useful.

Assorted other types of magic.

Thrilling!  Now, let's get a little more specific.  When you visit a page by entering its address into the browser, or by clicking on a link, that is a GET request.  When you submit a form with data to give to the server, that is a POST request.  The params hash is set up to capture the values provided by the user during these requests so that they can be used in our programs.

Returning to the ubiquitous Twitter example, let's say that Twitter has 500 million users because we like big round numbers (and also because that's how many users Twitter had as of 2012).  Twitter has not written code for 500 million unique user profile pages or URL end points, because that would be crazy.

So, what have they done?  Twitter, and many, many other websites have made templates for their sites, which are dynamically populated with unique data.  This means that my Twitter profile page:

Looks largely the same as Things on My Dog's Twitter profile page:

The layout is generally the same.  Certain fields are populated with our unique data:  our names, recent tweets, background pictures etc.  But those specifics have been added to a generic template.  

The params hash is a web developer's way of capturing the unique data and applying it to a generic template in order to dynamically generate webpages.  In the codebase for a website built using Sinatra there might be a block of code for dealing with GET requests for the path example.com/users.  This page could be a list of all users for the site, and by convention this page would have its code contained in an index file, such as index.erb (if you're using embedded ruby).  

But what about for all of example.com's 9 billion users?  They would all have a URL following the format of example.com/users/:user_name , and would have a corresponding block of code to program the behavior of those pages.  Each of those pages would be based on a template named show by convention.  That's show.erb if you're using ERB.

Examples are good, let's run with the above assumption that we're using Sinatra, and write out those blocks of code.   

Here we have our first use of params!  We have created a block of code that specifies the state and behavior for GET requests made to URLs following the format of example.com/users/:user_name.  What we define within that block will affect our show.erb file.  

In this case, the params hash will store the value of whatever user name is specified as :user_name.  How does one specify a user name?  There are a few ways, but in essence, any way you would conventionally visit a web page:  by entering an address into the browser or by visiting a hyperlink.  

Given a data structure containing all of the user names we can iterate through our list and conditionally produce unique links for whatever user we need.  Or perhaps the user name is entered into the browser.  Both are GET requests made of a specific user's page.  Whatever the means of specifying a user name, the params hash can collect the specific user name so that we can assign it to a variable. 

Building on this concept, imagine any social media site.  You have a list of users, each of whom has their own set of unique photos, tweets and settings.  If a user's pictures, tweets, or whatever other files are stored in a data structure for which a user name is the key, we can now pull up all of the appropriate unique data for any one user because params has stored a given user name for us.

To sum that whole mess up, params: 

Stores user's relevant browser input via URL's entered into browsers or visited via hyperlinks.

Stores user input submitted via HTML forms.

Dynamically generated pages use lists of these links (sometimes created by iterating over available data) and known valid URL endpoints to customize their generic template pages.

A cool application of params we recently applied at Launch was to store user input from various forms and perform client-side validation on those stored values.  We could check, for example, that no forms were blank or that any URL's submitted were valid, by storing each form's content as a variable and setting up the appropriate logic to check each of those variables.

Params is an incredibly useful tool once you understand it, and one that every aspiring developer should take the time to investigate properly.  Thanks for reading!

More from On and On coming ....soon!

[murphydbuffalo]

Don't fear the params hash! Part 1: An introduction to HTTP

Happy Memorial Day readers, I hope your long weekend was filled with shenanigans and tomfoolery.  If you've been hungrily awaiting more web development goodness from On and On you're in luck!  Today we have two blog posts.  The post immediately following this one discusses the glorious params hash.  The main conclusion of that post is that params turns out to actually NOT be powered by black magic.  Who knew?  

This post is a short explanation of HTTP aimed at beginners.  Possessing a basic knowledge of HTTP is key to grasping the concept of params, and my hope is that future Launchers can read these two posts and walk away with an understanding of what params is all about (and avoid the feeling of utter confusion that I had for the longest time).  Non-geek readers, you have my apologies.     

Being knowledgeable about HTTP is central to the role of a web developer.  If you are largely unfamiliar with the protocol, definitely continue researching the topic until you are comfortable with your understanding of it.  Here are a few great starting points, including some lovely diagrams from the early 90's, courtesy of the World Wide Web Consortium (w3), Code.tutsplus.com, and the HashPHP.org wiki.

In short, HTTP (Hyper Text Transfer Protocol) is the system of conventions by which clients (browsers) and servers communicate.  Clients request information from web servers, which are programmed to return appropriate responses.  Each HTTP request is comprised of a method and a path.  There are many HTTP methods, but the two most important varieties are GET and POST, from which all other HTTP methods are derived.

A GET request, as the name implies, is the browser requesting information.  To view a particular webpage, a GET request is sent.  This is typically done by entering a URL into a browser, by clicking on a link, or by submitting a form with a search query or login information.  

Conversely, POST requests allow the client to send data to the server, rather than receive it.  A post request is typically made by submitting a form containing data you expect to be persisted (stored in a file or database) by the server and possibly displayed on the webpage.  A GET request would allow you to view all of the tweets on your Twitter feed, while a POST request would allow you to post a new tweet.  Check out the example HTTP request below (from the Code.tutsplus.com link above):  

On the first line you'll see the method type, GET, followed by the path of the page being requested and the version of the HTTP protocol being employed.  Subsequent lines contain details including information on the browser  (under User-Agent), the type of data being expected in the response (JSON, XML, CSV, PDF, etc.), and the expected language of the response.

The next image shows the server's response to the above request.  Here you'll notice an HTTP response code in the first line.  It's the 3-digit number, 200 in this case: 

There are a variety of HTTP response codes, ranging from the 100's to the 500's, and you're likely familiar with at least a few of them.  Response codes in the 400's indicate a user error, such as 404 - File Not Found, meaning the user requested a path that no longer exists or doesn't contain the information being requested.  Response codes in the 500's denote a server error while a code of 200 indicates the request was fine and an appropriate response has been sent.  

Following the response code is more information on the nature of the response, including the date of the most recent changes made to the files.  Data about the nature of an HTTP request or response itself is known as the head of the message, while the content to be displayed or posted is contained in HTML form in what is called the body .     

There you have it, a look at the fundamentals of HTTP!  Hopefully this article has given you the foundation and resources to continue your progression towards being a great dev.  With these concepts in mind we're ready to tackle another important concept for any developer looking to use Rails or Sinatra, the params hash.

More from On and On coming ....soon!

[murphydbuffalo]

Vive la révolution!

Welcome back, dear reader.  The first week of Launch Academy's Summer 2014 cohort (the final 10 weeks of in-person problem solving and project building) is now history; one chapter in a book I hope to continue writing for many years to come.  History was always a favorite subject of mine in grade school.  Some kids found it boring, and sure, there were those periods of slow, plodding changes.  Man learns, over many years, how to plant crops and settle down in one place for a while - evolution.  

Evolution is the norm and evolution is great.  But what's really fascinating to me about history are those periods of rapid, sweeping changes.  Revolution!  Defying traditional systems of rule, reshaping social norms, introducing ideas that will manifest over centuries.  Washington, Robespierre and Marie Antoinette....those are the most compelling stories, right?  And in the history of my programming education, now is undoubtedly a time of revolution.

As summer approached I began questioning my decision to attend Launch.  Is it worth the money?  Worth the time and effort?  Is this the best way to grow into a well-grounded programmer?  The answer to each is a resounding yes.  So, in this one week, what has been so revolutionary?

Launch Academy has done fantastic job of bringing a truly diverse group of smart, motivated people together and pushing them to solve problems.  The most important things that we've learned in our first days here have not been new Ruby or Git syntax (although there's been plenty of that), but rather how to approach engineering challenges.  Have a tough problem that's making your head spin?  Don't you dare touch that keyboard.  Grab the whiteboard and diagram what needs to happen.  Use flow charts, and use plain English.  Now, pseudocode.  What are the end user's needs?  What smaller parts can we break this down into?

Think you've wrapped your head around the problem?  Good, but how well do you explain it to others?  Can you go up to the appropriate level of abstraction based on your audience?  Can you practice patience and put your ego aside in order to learn another, maybe better, way apart from what's comfortable to you?  Of course these aren't shocking concepts, this is common sense.  But common sense isn't always common, and the magic of Launch is in ensuring the implementation of these techniques.  It's one thing to preach the virtues of this flavor of critical thinking, but it is another thing entirely employ it out of necessity.

In our cohort's first week nearly everyone has put in 5 consecutive 10-12 hour days, and while we may end the day drained, enthusiasm abounds the next day.  Case in point, I'm now sitting at Mission Control writing this post on a beautiful Saturday afternoon with a room full of fellow Launchers, marked up whiteboards and code-filled text editors.  There is no requirement that we be here on weekends, only our desire to get better.  We are struggling together, learning from one another, and becoming practiced at applying these methods to challenging programming puzzles.  We've heard firsthand, from senior engineers like RBM's Johnny Boursiquot, about the importance of these problem solving and communication skills.

This is the value of Launch Academy; immersing yourself each day in not only the technical knowledge of web development, but in the mindset of pragmatically dissecting and producing solutions for problems.  Those are the abilities that employers want.  To know one technology or another is an asset, but understanding how to analyze, solve and communicate effectively is invaluable.

More from On and On coming ....soon!