Jake Does Data

Visit me on LinkedIn!

[jakedoesdata]

Figure 1: We have an image of Derek Zoolander that has been corrupted.  Using Fourier transforms and a Gaussian filtering technique, we will experiment with various filter sizes to see if we can clean up the image.

Figure 2: The logarithm of the Fourier transformed image to show the frequency content of the original image of one of the color channels.  The original image is RGB which has 3 channels (one for red, one for blue and one for green), so our technique requires us to separate each channel, transform it, filter it, and put it back together.  The channels are all very similar as the noise is essentially normally distributed across each channel equally, so you can imagine the other two by just seeing the one.

Figure 3: The filter we will use.  Two dimensional Gaussian filter, very standard for EDA and a great place to start.  We take the transformed content matrix, and multiply (element by element) by the filter matrix.  What happens when we do this?  The maximum value in our filter is 1, the minimum is 0.  Notice in our transformed image (figure 2) that most of the content is centered in the middle.  We take our filter, center it over that cluster of data, then multiply.  It then keeps the majority of the center frequencies, because the filter at that point is either 1 or close to it.  The filter then decays toward the edges, and sets the rest to zero (at the same that the Gaussian decays).  

Figures 4 & 5: Results with varying success.  The Gaussian does a great job of blurring the image, but doesn't really bring a whole lot of clarity to the final result.  

Figure 6: Another color image of Derek, this time with a very specific area of corruption.  We will use the heat equation to diffuse this specific area, bringing clarity to the corruption.

Figure 7:  The heat equation (http://en.wikipedia.org/wiki/Heat_equation) we will use to diffuse the image.  The heat equation says that heat diffuses through time into neighboring less "hot" areas, reaching an equilibrium.  We can use this method to see image values (that are really just represented in a matrix as double precision values) as "heat values" allowing them to diffuse into neighboring areas, mixing the noise.  For our specific case, we will use the ODE45 command in Matlab to run our simulation with a right hand side equation function that we build and pass in.

Figure 8: We need to narrow down the area we wish to diffuse, so we experiment with matrix indices to find the correct area.  We don't want to diffuse the entire image, and sending that much data to ODE45 is unnecessary. 

Figure 9 & 10: Results with varying success.  The main variable we tweak here is the time stepping.  How long do we let the simulation go before we stop it?  The longer it goes, the more time the diffusion has work, but too long can cause weird boundary anomalies, which will lead to a rough blend into the diffused area.  For spot treating in areas where the results require a uniform texture, this is a fairly successful approach. 

[jakedoesdata]

Summary: Using the Fourier transform for analyzing audio exposes a major weakness with this incredibly powerful tool.  When transformed, a signal loses its entire time domain.  In order to find which notes are played at which time, we need to discretize and filter our time domain, giving only small pieces to the Fourier transform to work with.

Window size is very important here.  The Fourier transform doesn't care what you give it, it will transform the entire chunk and display the frequency content of it.  If it includes too many notes, then the frequencies will be displayed as such.  For clarity in the time domain (the wider the window, the more frequencies will bleed through) we shrink the window.  Here is another crucial issue however: waves have a frequency and length, therefore they are not independent of window size.  The smaller the window, the higher the frequencies that will pass through, and vice versa.  This means that, for what we gain in the time domain, we lose in the frequency domain.  Heisenberg's Uncertainty Principle at work.

Figure(1) Windowed Fourier transforms analyzing "Hallelujah Chorus" from Handel's Messiah.  

Figure(2)  Discretization and filtering of Hallelujah.

Figure(3) Discretization and filtering of Mary Had A Little Lamb on recorder.

Figure(4) The spectrogram of MHALL, showing a musical staff of the frequencies.

ALL FIGURES PRODUCED IN MATLAB.

[jakedoesdata]

[jakedoesdata]

Hypothesis testing and analysis of Madden running back data:

I thought my results for the Madden cover running back data were pretty interesting so I decided to test the hypothesis that running backs who were on the cover of Madden performed worse the year after making the cover.  

Null hypothesis: a null hypothesis is a claim that you revert back to when the data you've collected fails to make a compelling case to the contrary.  In our case, the null hypothesis represents the claim that there is actually no difference in the performance of running backs that make the cover of Madden vs running backs who don't, with respect to the year they make the cover.  

Alternative hypothesis: This is the claim we default to when the null hypothesis has been rejected.  In our case, the alternative hypothesis is the claim that there actually IS a difference in performance.  

Choosing which statistics to use: I decided to look for which year running backs made the cover of Madden in their career, and average those values to come up with a base line value to collect data against.  The value I came to was 6.25 years into a running backs career is when the average Madden running back will make the cover.  I then gathered 22 other running backs data.  I took their 6th season and 7th season total yard stat and then took the difference.  There is in fact a downward trend between a running backs 6th and 7th season, with an average loss of roughly 140 yards (no where near the average Madden cover running back loss of ~910 yards!).

Complicated mumbo-jumbo: The value above (t) is the 'critical t value' on the x-axis of a t-curve.  We take that value, with another calculated value (degrees of freedom) and look up in a table what the probability is that the data we're testing differs by "pure chance." What does this mean?  Long story short, it means that the probability that there is no difference in the population mean values is essentially 0% given the t value we calculated.  4.1 is very high.  

Take Away: There is a very small chance (with the data gathered) that the difference between the mean values is zero, with a P-Value = 0.004 (there is a 0.4% chance that the difference is pure chance). Madden cover running backs perform worse.

[jakedoesdata]

I charted running backs (from the Madden cover) total season yardage to see if there was a trend to indicate a Madden curse.  There is a considerable drop off after a running back makes the cover, HOWEVER, this is not evidence of a curse.  Take a look at the left side of the graph.  It looks as if there is a "build up" before the madden release.  This seems like evidence to suggest that the cover is a better indicator of a players peak year, not necessarily a curse.

[jakedoesdata]

Goal: I'm going to restate this a lot, because I think it's important to have in the forefront of your mind when interpreting the data.  My goal is to take image data sets and trim the amount of images such that humans only need to analyze a fraction of the whole set when searching for anomalies amongst the images.  This saves time, money, and most importantly, it can potentially save lives.

A word about machine learning and linear discriminate analysis:

Below (previous post) you'll see a blue and red line graph of two normal looking curves.  Those lines represent the frequency (how often something occurs) of values that my algorithm has given to images of satellite data.  You can think of these values as the "probability this image contains an anomaly", and how those values are reached is explained below (hint: it uses a training set to teach my computer through machine learning which images contain anomalies).  Since these curves are normal, it means that we can run a great deal of analysis on them, particularly linear discriminate analysis.  

Here's the idea:  I choose a bunch of images, and decide for myself if these images contain or don't contain anomalies.  I mark "does contain" with a 1, and doesn't contain with a 0 (I literally created a hits and misses 1's & 0's array for this).  Then, I allow my algorithms to assign values to these images based on the criteria in the previous post.  Now I have two numbers assigned to each image.  A 1 or 0 representing a yes or no (respectively), and some value the computer has chosen (these values range from approximately 2.71 to 49.12, not really relevant, but just to clarify that these are not discrete values, they are real number representations of the probability and image contains an anomaly).  Then, I do what is called linear discriminate analysis on those values.

Linear discriminate analysis looks at those two values, 1 or 0, and the real number representation of the probability an image contains an anomaly, and builds a function.  To get a feel for what it does, scroll down to the blue and red line graph from the previous post and imagine a vertical line drawn somewhere where both curves overlap.  Depending on the location of this curve, all satellite images that land on the right are flagged as a hit, and all values that land on the left are marked as a miss. 

Now we have an LDA (linear discriminate analysis) function.  So instead of finding new data, I run my test set back through, and have the LDA function predict out of that image set, which images contain an anomaly.  With those predictions, you get the graphs above.

Interpreting the graphs:

Figures 1 & 2 legend explanation: 

     False Alarm: Photo does not contain anomaly, but was flagged.

     Complete Miss: Photo contains anomaly, but was not flagged.

     Correct Hit: Photo contains anomaly, was flagged.

     Correct Miss: Photo does not contain anomaly, was not flagged.

Figure 1: Here we see two prediction sets, and their successes based on minor optimizations. Instead of running the basic probability values through my LDA tool in R, I chose to run the log base 10 of those values.  This is because the "misses" data shows somewhat of an exponential distribution when we don't take the log of the values.  Taking the log of the values transforms the data to be more "normal" without sacrificing it's predictive quality, and also allows to do sweet things like linear discriminate analysis.

Figure 2: This is technically the most important part of the experiment.  Getting the highlighted value down to zero is extremely important because those are images that contain something, and didn't get flagged.  This is a problem.  The only thing we don't want flagged are images that contain no anomalies, which isn't yet happening based on my algorithms.  I'll be working on this.

Figure 2 & 3: Pretty self explanatory, just percent of successful finds and misses.

THE TAKE AWAY: So far, I have demonstrated that my algorithms can successfully trim an image data set from 114 images, down to 46 with a success rate of about 88%.  This is good, but we can do better.

[jakedoesdata]

This is my preliminary data for my airplane wreckage debris finder, and this is very good news.  Let me explain my process:

First, my goal was to create an automated process for narrowing down images that required searching by humans after the Malaysian flight disappeared over the ocean.  Many images were needlessly searched by humans, and I wanted to decrease the work required.  This is actually seen in the real world by places like Microsoft, who use similar tactics to narrow down the legal documents that are required to be sent to their legal team for discovery when they are being sued.  If they can narrow down the documents that their lawyers have to wade through, their legal costs go down.  Very cool.

Next, I have a training set of data.  I've collected images from the internet of the ocean, some showing anomalies, some showing nothing.  I then mark the images I want flagged, and the images I want the algorithm to ignore.  In the graph above, the red line represents images I want ignored, and the blue line represents images I want the algorithm to keep.

Next, I run these images through a Matlab script that turns the image into a black and white binary image, which is that image above that looks like outer space.  The black and white image is showing things that stand out (white) vs. the background (black).  Next, I lower the threshold or sensitivity, so that it starts only picking large anomalies.  I divide the area of the anomalies by the number of anomalies to get the value charted on the x-axis of my chart above.  The larger that number gets, the higher the likelihood of the image containing an anomaly.  

Here is the reason a normal distribution is good:  I can use linear discriminate analysis to find a value which "everything above" is a hit, and "everything below" is a miss.

GREAT NEWS!

[jakedoesdata]

Charting BPM and song length over time on the Billboard top 100

[jakedoesdata]

Had some fun with some NFL data I ran across.