As detailed previously, we were going to build a health app in 48 hours. I got to searching through Canada's data, and stumbled upon the Nutritional Value of Fruits and Fruit Juices. The data was billed as a "list of 19 nutrients for 1000 of the foods most commonly consumed in Canada". And yet, there were only 106 foods in the data, with no link to other datasets from the same group. Inexplicably, this took several hours to figure out and find the accompanying data. So, around noon on Saturday, we finally started. Hold up.
Saturday? The Hackathon started Friday, at 13:00PST. Alan had to go to the other university campus for something, I had work until 9, and Matheson was the most productive and started figuring out how to use the data. The rest of Friday night was spend trying to understand Core Data, which none of us had any prior experience with. And then I went to a bar.
I woke up, and the Hackathon was already 24 hours in. In those first 24 hours we realized we were in way over our heads: we had never had to deal with a database before, much less populate Core Data from a dozen CSV's. I jumped into RStudio, an awesome IDE for the R statistical language, and tried at least getting one large table from all of these smaller, food-group specific tables. I was thrilled to start using the data, and even tweeted about it:
Already found the perfect data set. Beginning the perfect app to go along with it. #CODE2014
— Alex Popov (@TheAlexPopov)
There was no consistency between the CSVs, that were organized by Canada. Let's take a look at 2 examples and how badly they set us back.
We have the Fruit table on top, and the Baked Goods table underneath. A large problem is the lack of a consistent Measure column. At least in baked goods it's always comparing singular units, not "6 halves" to "3 wholes" of apricots (which incidentally, is the same). A much bigger problem is how one has a row for measurement units, while the other does not. Simple as this is for a human, a sqlite3 database can only have one set of headers, and then columns of data underneath. R, while much more flexible, also has no logical way how to interpret this. These are just two of the tables. They all had their particular quirks, and would have to be adjusted. By hand. As a programmer, I never want to have to repeat any action more than I absolutely have to.
I spent about 4 hours trying to solve this problem. There were a variable amount of columns, including contextual "nutrients" such as alcohol and caffeine where necessary. With no real data to drive the app, Matheson couldn't start coding anything beyond four squares as a home screen that would go to the Big Four Features we had originally expected to implement.
The solution was surprisingly simple: if you can't cook with the crap you're given, get better ingredients (I worked as a night-shift cook at Denny's, but that's a post for another day). My father was gracious enough to take some time out of his day to find us a much more complete and systematic data set from Health Canada. This data was amazing! It was modular normalized; with separate tables for pretty much each piece of information; food and food groups were separate; measurements were all in 100g portions and even had another table that acted as the conversion factor to find amounts in silly units like "6 halves". Despite adding to the learning curve, we finally had what we need to start writing code. And write code we would: by this point we only had about 16 hours left.
What did we do with those 16 hours? That (and actual code) will be in the next post.
Open Data failed
Tired of cooking with this crap
Dad threw me a bone
