Noah Kahan
No title available
PR's Tumblrdome
cherry valley forever
shark vs the universe
almost home
noise dept.
Kiana Khansmith
No title available
roma★
Cosmic Funnies
todays bird
bliss lane
No title available
pixel skylines
The Stonewall Inn
Interview Vampire Daily
No title available
★
KIROKAZE
seen from Colombia
seen from United States
seen from United States
seen from Kenya
seen from Colombia
seen from United States
seen from Bangladesh
seen from Italy
seen from India
seen from Brazil
seen from Iraq
seen from United States
seen from United States
seen from Iraq
seen from Israel
seen from Sweden
seen from Bangladesh
seen from United States
seen from United States
seen from United Kingdom
@stevedoesdatascience
<a href=“http://yann.lecun.com/exdb/mnist/”>MNIST</a> is one of the more famous data sets in machine learning. It was developed to help train image processing and handwriting analysis programs. It’s the basis on which a lot of automated check processing systems were originally built.
The World Hunger Map, describing the progress that has been achieved since the early 90s, in eliminating or reducing hunger and starvation.
Scikit-Learn is a fantastic Python library for Machine Learning, Data Mining and Data Analysis.
An interesting look at how threat perception has changed in the last half-decade. Conclusion, U.S. citizens have become a lot more frightening. (source)
CS109: Lecture 10 (Review)
As I mentioned here, I'm trying out a new format for my lecture reviews. Let me know your thoughts. This is a review of lecture 10 of Harvard's 2014 introductory Data Science course. 2:00 - This lecture is about regression and baseball! 5:00 - Now we are going over the rules of baseball, because statistics. 8:00 - A lot of stats that can go into a baseball player's "fitness" score are obvious (e.g. Home Runs). The trick to optimizing a team on a budget is to find non-obvious stats that still denote quality but may not be quite as well known. 15:00 - We are still talking about how baseball works. 18:00 - One of those hidden (cheap) statistics was "base on balls," the measure of how many times a player is walked. 20:00 - Yes! Regression. 23:00 - Interestingly, batting average seems to be normally distributed. 26:00 - A person's rookie year batting average is predictive of their sophomore year batting average, fortunately for baseball, batting average doesn't seem to be entirely luck. 33:00 - "In simple regression you presume the the data is bivariate normal and that any strata it's also normal." The motivation for this was seeing that the averages of the box plots of the freshman vs sophomore batting averages of baseball players fits a line. I knew we were going somewhere with this! 37:00 - Explaining how the regression line formula works. 43:00 - Since the correlation between batting average year to year is not 1 (it's more like .41), we would predict that someone who is several standard deviations over the mean one year, will do worse the next, and someone several standard deviations under, will do better. Regression to the mean, as the kids say. 53:00 - While "base on balls" is predictive of runs scored, it's not a stat that used to be highly valued. You can buy a player with a high BoB for relatively cheap. 1:00:00 - In the real world, sometimes you have to make a prediction at a current time step when you only have data for precious time steps. Unfortunately, when luck is involved, that can reduce the strength of your prediction. The example they used was in baseball, if you're trying to predict whether a player will generate a lot of runs, the previous year's runs, especially if they were very good or very bad, may not be a perfect predictor -- due to mean regression. 1:05:00 - Q&A time! Overall Rating: 2 of 3 (scale) Discussion: I like the Baseball metaphor quite a bit, and it's a useful way of discussing regression. Moneyball is also a good example of how Data Science can seperate itself from more narrow applications of pure statistics.
Methodological Change
I’m changing the way I review stuff!
I’ve decided to take notes as I watch lectures/read chapters rather than trying to remember them afterward. Not only is it easier on me (a big plus), it’s also a more accurate representation of my feelings in the moment.
The point of this whole exercise is to chart how much I (a data-oriented professional-type-person) feel I’ve learned, and this more “real-time” approach is, to my reckoning, a superior way of getting at that.
Speaking of charting, I’m also going to start rating each lecture on a scale of 1-3. That’s about as precise a rating as I think I’m capable of. The scale will breakdown as follows:
Rating 1 - Mostly review of what I believe your average person would have learned in an introductory statistics/CS/machine learning course. For beginners only.
Rating 2 - Information is presented that either combines statistical and CS concepts in an novel way, or provided me with a new way of looking at an old concept that was specific to Data Science as an independent discipline.
Rating 3 - Entirely new information driven either entirely by Data Science specific concepts, or statistics/CS/machine learning concepts that have been made more interesting by looking at them through a Data Science lens.
CS109: Lecture 9 (Review)
It has been a while since I’ve done one of these. There are a lot of reasons for that, including but not limited to: making a giant spreadsheet so I can buy a new car, working on client things, getting ready to pick up and move my life 200 miles south, and pure laziness.
Anywho, I finally got around to watching lecture 9 of Harvard’s Data Science course, and here are my thoughts as I had them:
1. We started with a review of variance, standard deviation, mean, random variables and associated equations.
I may or may not have fallen asleep for some portion of this.
Remember kids, the mean of the sum is the sum of the means.
2. Alright, alright,I should stop being cynical -- one of those rules I will admit to not remembering shows up at about minute 16:
"If independent, variance of sum is sum of variance."
Still...
3. Polls often suffer from the "House" effect, where the pollsters themselves introduce systematic biases which cause their results to be incorrect consistently.
Instead of using the data itself, treat their results as random variables, and take an average of the average.
"A poll is a much better predictor of an election than a person."
4. Loess data smoothing is based on Taylor's Theorem, if you window your data small enough, you will always end up with a line.
Just keep fitting lines to your data until you end up with a data fitting curve.
5. The Nate Silvers of the world often take polls, weigh them by their error rate, sample size, etc... And then takes the average of averages as before.
More generally, the latter half of this lecture is far superior to the beginning. It even had a fun review of one type of data smoothing, which I found to be really useful.
I would definitely watch this again, but would probably skip to minute thirty to get to the read meat.
Three Things: Exploratory Data Analysis
Back to reading Data Science: Straight Talk From the Frontlines, this time the section was on Exploratory Data Analysis. Most generally, Exploratory Data Analaysis is the art of taking a set of data, graphing it in various simple ways (histograms, scatterplots, box plots, etc...) and trying to get an initial feeling for what the data might be telling you. It's what you learn to do at the very beginning of any statistics course you might take, here are three interesting things I learned about it from the book. 1. EDA is not just for Intro to Statistics 101 One of the things that differentiates Data Science from applied mathematics, are the types of question each field is called on to answer, and the philosophies that result from each. While statistics often relegate EDA to subordinate role, necessary but not particularly interesting, Data Science uses it drive much of it's subsequent work. 2. EDA is about intuition In Data Science you might be working with the same set of data for a long, long time and it's important to gain some intuition on how the set works, if you're going to make reasonable predictions based on it. EDA is extremely useful for this. If, for example, your corpus is all the tweets in the world, you can use a histogram to know which days have the heaviest traffic or a scatterplot to see what terms tweets cluster around. It also is your first line of defense for discovering outliers and sanity checking data collection. 3. EDA helps you understand what you're measuring. If your job is to create an algorithm to determine how "popular" something is relative to another something, it helps to know what "popular" means in terms of the data. EDA can be very useful for this, providing a simple way of understanding how the data is "shaped" and start developing a sense for what an abstract idea like "popularity" might mean in your particular case.
Examples of some common statistical distributions.
Three Things: Statistical Inference
This morning I was reading the next chapter of Data Science: Straight Talk From the Frontlines, and it turned out to be a pretty good review of statistical inference. Broadly, Statistical Inference is the process of using data to determine properties and answer questions about populations. As an example, when we try to predict how many people think a politican is doing a good job, from a sample of the population, that's a statistical inference. We can't poll the entire population, but we can get a pretty good picture by looking at a subset. Back to the chapter! Here are three things I took away from the chapter vis a vis, Data Science. 1. Just because you have "Big Data" doesn't mean you have useful knowledge. No matter how complete a data set you think you have, it's likely to have biases'. The example they gave was hurricaine Sandy and Twitter. If you just looked at Twitter data at the time (a huge corpus), you would think the hurricaine wasn't all that bad, however, it would ignore the fact that those people who were really struggling with the hurricaine, were probably not on twitter. 2. Data Science requires being comfortable sampling from a variety of formats. I see this all the time, modern data analysis involves a pile of "messy" formats: images, sensor data, tweets, emails, HTML-laden text, etc... It's important to know how to deal with them and the different biases' they introduce. 3. The Four Vs of "Big Data" Volume, variety, velocity and value. This is a useful (enough) way to think of "Big Data". Data can be considered "big" if there is a large amount of it, in multiple flavors, coming in quicklyl and most of it has business value. In this case the weasle phrase is "large amount," which can only be defined relative to the time when you're doing the analysis. In 1994, a gigabyte of data would be huge, these days it barely registers.
What Is A Data Scientist?
I started reading Doing Data Science: Straight Talk From the Frontline in earnest this morning, and the first question it poses is, “what is a data scientist?”
While the book does a fantastic job of exploring the answer, it has been a few years since it was written, so I thought it might be fun to take a crack at it myself, from the perspective of a professional data dabbler.
For my money, “data scientist” can most usefully be defined as, “a professional with a background in statistics, machine learning and applied mathematics who uses the tools of computer science, design and data visualization to answer complex numerical questions.”
The nature of those questions, the size of the data sets, and the additional pieces of domain expertise that this professional brings to the table seem secondary to this basic interplay between knowledge and tools. Thoughts?
Erin LeDell, Data Scientist & Hacker at H2O.ai H2O World 2015, Day 1
…with a shout-out to yours truly :) Thanks Erin!
What is machine learning? See how it works with our animated data visualization.
I one day hope to make something half this cool. This is an introduction to some of the underlying concepts of machine learning, using a fantastic visualization.
In machine learning, computers apply statistical learning techniques to automatically identify patterns in data. These techniques can be used to make highly accurate predictions.
Keep scrolling. Using a data set about homes, we will create a machine learning model to distinguish homes in New York from homes in San Francisco.
The official stereotype lab of Yanko Tsvetkov, a bestselling author, prolific cartographer, and leading international bigotry professional with a taste for salacious political incorrectness and unconventional historical studies. Opinions are not his own, he’s merely a curator :)
I’m going to file this under unconventional data sources:
The official stereotype lab of Yanko Tsvetkov, a bestselling author, prolific cartographer, and leading international bigotry professional with a taste for salacious political incorrectness and unconventional historical studies. Opinions are not his own, he’s merely a curator :)
Polls, charts, forecasts and data about upcoming elections, Obama, Congress, Democrats, Republicans, politics, health care and the economy.
A source for public polls and election data. We used this in lecture 8 of CS109.
CS109: Lecture 8 (Review)
This entire lecture was about polling!
We went over random sampling, monte carlo simulations, binomial distributions, QQ plots (to test if a distribution is normal)
and ways to make more confident predictions when doing random polling (either widen your confidence interval or take more samples).
There was also a bunch of code showing how to do all of this in Python.
The lecture ended with a discussion of some ways that real world data collection (see: political polling) can differ from sampling from a less “messy” set. Things like time to election, lying and sampling error can skew values substantially.
Overall, it was an interesting lecture but really geared towards someone without a strong statistical background -- except for the Python stuff -- which is always nice.
This is the library we’re using in CS109 to do data scraping. We are also borrowing a little horse power from urllib.
