TIL

Things I learn at my crazy data science job. Written for my own knowledge persistence, though maybe you'll learn something useful along the way as well.

[sarguido]

I have a blog?

I have a blog!

Update: I no longer work for Reonomy. Now I work for Bitly.

Another update: I want to blog more. Hmm.

Here's a round-up of links I've been tweeting about:

http://journalismcourses.org/D3.html Awesome-looking data visualization class taught online.

https://medium.com/coder-snorts/intro-to-python-s-pandas-7a52eefd3313 A nice intro to Pandas by a DC PyLady

Both PyData Paris and PyOhio have CFPs open: http://pydataparis.joinux.org/#cfp and http://www.pyohio.org/call-for-proposals/

Slides from my 'Analyzing Data with Python' talk: http://www.slideshare.net/SarahGuido/analyzing-37297441

That's it for now. Keep it real, keep it Pythonic.

[sarguido]

ggplot for Python

Dear nonexistent readers and Tanya, I have a confession to make, one that I've tried to avoid making for a long time.

I HATE matplotlib.

There, I've said it. I despise it. It's the least user-friendly Python module I've ever worked with. It's ugly, hard to customize, and unintuitive. Though matplotlib is a big contender in the scientific Python visualization arena, I've always tried to avoid it for the reasons I've said above. 

Fortunately, there's a better way. Enter ggplot for Python.

ggplot is gifted to us from the amazing company Yhat. It's essentially a wrapper around matplotlib that renders graphs as one would in ggplot2 in R. Both libraries are based on The Grammar of Graphics, which isn't something I've read but probably should. 

ggplot works the same way in Python as it does in R: to create graphs, you basically layer things on top of each other. While it's not the most customizable visualization library you're ever going to work with, it makes generating exploratory plots SO MUCH EASIER and much more nicely. From the documentation:

The result is less time spent creating your charts, and more time interpreting what they mean.

The best part of ggplot? You can pass in Pandas DataFrames. HOLY SHIT! Sign me up. My love for Pandas is unbounded.

I'm going to actually provide a code example of how this works, but I need to figure out how to do syntax highlighting/code blocks in Tumblr. So, to be continued...

[sarguido]

Midwest.io 2014 - Building a Production Machine Learning Infrastructure - Josh Wills

This talk was given at Midwest.io 2014. Cloudera's Data Science Team has a simple mission: build an analytics infrastructure so awesome that it makes Google'...

Interesting and a great talk.

[sarguido]

Gini impurity vs. information gain

In the context of decision trees/random forests:

The Gini impurity (not to be confused with the Gini coefficient, which is a measure of income inequality) is a measure of how frequently a randomly chosen observation from a dataset would labeled incorrectly, if it were to be labeled according to the distribution of labels present in the dataset. It's computed by summing the probability of each item being selected multiplied by the probability of a mistake in categorizing that item. 

On the other hand (and, apparently, at the risk of disappearing down a rabbit hole), information gain is based on the concept of entropy from information theory. Entropy is the average amount of information contained in a message and therefore gives us a way to think about uncertainty about the source of information. When we say 'information gain' we are referring to the Kullback-Leibler divergence (basically the divergence of one probability distribution from another) and sometimes even mutual information (a measure of features' mutual independence).

So how does that relate to machine learning? Basically, information gain is used to narrow down which features lead to a certain class label. The above paragraph makes it sound way more complicated than the concept actually is.

(One of my major gripes about many explanatory machine learning/mathematical texts is that they're written by experts for experts. Us laymen get left behind, or end up writing blog posts like this to puzzle through the jargon and piece it together. Also why I'm writing that book that has yet to materialize.)

Questions that I don't know the answer to:

As far as I can tell, the Gini impurity is only useful in conjunction with decision tree learning/random forests. Can it be used elsewhere? Would it be useful elsewhere?

Do I need to know more about information theory? I went to grad school in a program called information science which, according to Wikipedia, is completely different. Would knowing more about information theory help me understand more about machine learning?

Has anyone else on Tumblr tagged their post with 'gini impurity', or am I the only one?

Update: I am the only one that has tagged a post 'gini impurity'. :(

[sarguido]

Providing clarity to the world's most opaque markets.

<3

[sarguido]

Shell commands

Here are some fun shell commands:

head <filename> : see the first 10 lines of a file

less <filename> : load the file one page at a time (not all at once in memory)

cat <filename> : print out the whole file

wc -l <filename> : count the lines of a file

sed 's/<string>/<replacement string>' : replace something with something else

Keyboard shortcuts for Mac:

Ctrl + a : jump to the beginning of the line

Ctrl + e : jump to the end of the line

Ctrl + k : delete from wherever the cursor is

[sarguido]

Debugging in Python with ipdb

Today I learned about ipdb, a tool that is going to make my life so much easier I could cry. ipdb is an interactive debugger for Python. You can set break points and step through code like you can in an IDE. 

Commands: https://docs.python.org/2/library/pdb.html#debugger-commands

Have I actually used ipdb yet? Nope. But I will.