Fixing the Google Chrome 'Aw Snap' error
Only kick-ass developers work at Darwin. And instead of keeping our knowledge to ourselves, we'll be sharing our development adventures on our blog. First in this series: a blogpost by Thomas Tuts, our Front-End Engineer.
Out of nowhere, a mysterious error blindsided us that only seemed to rear its ugly head in Google Chrome 32. This is the story of how we beat it — with lots of patience, perseverence and pizza.
How it all started
On a (probably rainy, knowing Belgium) Thursday morning a client notified us that our app no longer worked on his install of Google Chrome. Indeed, we were instantly able to reproduce the crash — two of the three views in our app were no longer responsive and simply showed an 'Aw Snap' page, courtesy of Google Chrome.
Now, you might or might not know this, but the devious 'Aw Snap' is a victim of its own simplicity; it simply tells you what went wrong, but not exactly what, or how to fix it. This is pretty frustrating as a developer, since it means you're kind of looking for a needle in a haystack.
The crashes created no usable logs, so we had to get logs from the source — Google Chrome. After some cursory research, the best approach seemed to be debugging the actual Chrome binary to see what exactly went wrong. Doing so didn't yield a lot of interesting results (at least for someone not well-versed in Chrome internals, i.e. me). All we could make out from the logs was that the renderer was crashing for some unknown reason.
The War Room
Since it became more and more likely that this wasn't a bug we'd be able to solve in an hour, we decided to huddle up in a War Room. In total there were three developers: two back-end engineers and one front-end engineer (myself). The major problem with the app is that it's currently being ported; the old codebase was extremely bloated and unmaintainable (legacy code that was built before any of us got here).
The first step was reproducing the error on a local environment. To do so, we used Vagrant (which was already set up, but not fully configured yet). After some Vagrantfile fiddling, we managed to get it up and running. Lo and behold, the same crash happened on our local environment. This was A Good Thing™.
We managed to narrow down the error to the following two lines of code:
var focusPeriodStart = moment.unix(funnelStartDate).format('YYYY-MM-DD'); var focusPeriodStop = moment.unix(funnelStopDate).format('YYYY-MM-DD');
The funnelStartDate and funnelStopDate variables were genuine UNIX timestamps that didn't seem to indicate any bug-worthy behaviour, yet they were clearly the culprits of the crashes. After changing the code to calculate the date from the UNIX timestamp with vanilla JavaScript, we managed to completely eliminate the error:
focusPeriodStart = new Date(funnelStartDate * 1000); focusPeriodStop = new Date(funnelStopDate * 1000); focusPeriodStart = moment(focusPeriodStart).format('YYYY-MM-DD'); focusPeriodStop = moment(focusPeriodStop).format('YYYY-MM-DD');
To be honest, we're still not 100% sure what the exact cause of the crash was, and we probably never will be. Apparently, some people online thought the call stack size of Google Chrome got reduced in the v32 update. This would explain why the crash only started happening on Thursday - v32 was released roughly the evening before.
Lessons learned
When an ugly bug shows up, don't try to take it on alone. Ask other people to help out. Among other things, you'll be able to act as a sounding board for eachother.
Get rid of an old, bloated codebase as soon as you can. Eventually, some mysterious bug will turn up. When it does, it'll be insanely hard to debug; especially if you're not familiar with the codebase at all.
Trust yourself and your team. Things were pretty rough, but I was absolutely sure that we'd get to the bottom of it. I have complete faith in my team and its ability to quickly cope with difficult situations. The hiring process at our company is pretty rigid, so you can be damn sure you're surrounded with some of the most talented people you'll have the pleasure of working with.
When in doubt, order pizza. Obviously, it doesn't have to be pizza — just be sure to take a break now and then to stretch your legs. Distract yourself for a little while, and you'll find that you and your team will come up with a solid plan of attack.