Data And Technology

On Statistics, Technology and Finance.

[eipico-blog]

Modelling Bias

By selectively choosing the data you can "prove" that Republicans have been better for economy or vice versa and the impact is "statistically significant". This site let's you do that in an interactive way.

Rising above our biases is hard.

http://fivethirtyeight.com/features/science-isnt-broken/

[eipico-blog]

What do we need to calculate VaR?

To estimate the probability of the loss we need:

The probability distributions of the individual risks

The correlations of these individual risks

The impact of these risks on the value of the asset/portfolio

 Examples of individual risks – interest rate changes, equity market volatility, economic growth and so on. 

[eipico-blog]

What is Value at Risk?

Before making an investment it seems reasonable to ask "How much can I lose on my investment?" . But if you really think about it the literal answer to that will be "As much as you invest (and possibly more if you use leverage". However that's an answer that is unlikely to convince any investor !

Instead the approach that VaR takes is to rephrase the question so that we are talking probabilities and time frames, so the definition I like is this:

"The Value at Risk measures the potential loss in the value of a risky asset over a defined period for a given confidence interval. "

You want to be able to make a statement like; there is a 95% chance that your portfolio will not lose more than 20% of its value in 30 days. 

[eipico-blog]

Bayesian Statistics!

[eipico-blog]

Data Munging: Identify Columns in CSVs

"Data munging or data wrangling is loosely the process of manually converting or mapping data from one "raw" form into another format that allows for more convenient consumption of the data with the help of semi-automated tools."

So now I have tens of thousands of CSV files with data about a lot of people and I need to identify the columns in the CSV. 

The first method that occurred to me is to look at the headers and try to figure out what they could mean. But even looking for something like email is not straightforward - there could be a header called email, email id, email_id and oftentimes the header is simply missing. So, unless your data is very structured (which it seldom is) this does not work. 

Emails and Phones

I identified email columns by saying that if more than 50% values in a column are valid emails then that column is Email. Similarly for phones. 

Names

This is more interesting. There is no regular expression for identifying a valid name. So here is what I did - I took a list of say 100 most common first name in the geography that the data was from and then said, if more than 10% values in a column are found in this list then that column is the first name and similarly for last name. Then, if the ratio between number of first names matched and number of last names matched is between 0.2 to 5 then it is simply the full name. Of course, this doesn't give me 100% accuracy but the results were pretty good and adequate for my needs. And it definitely beats going through all the files and manually identifying the columns!

This technique works even better for identifying cities and states and is useful in many other cases.

[techcrunch]

This comic is an awesome checklist for all future anti-tech hysteria Image via xkcd

[eipico-blog]

In God we trust. Everyone else, bring data

Michael Bloomberg

[eipico-blog]

Let customers suggest features!

Sounds like a no-brainer. But most websites do not have a structured way of doing this. At an online money management company that I worked with we used this interesting tool called Get Satisfaction. 

This tool lets you share ideas and also lets other customers "up-vote" the ideas that they see. We saw a number of ideas bubble to the top with this tool. So we already knew that there was an engaged audience for the features that we developed instead of hacking away at some obscure feature only to find that there we no one wanted to use it when the feature was deployed. 

You may or may not like this tool, but the idea behind it is powerful and I have used it in other situations as well - its a good way of crowdsourcing ideas. 

I am constantly amazed by the amount of development effort that is wasted on features that no-one wants to use. This is one way of reducing such effort. 

[eipico-blog]

Randomness "Coming Soon"

So, there are 3 possible product features that different stakeholders are championing. The opinion is divided, marketing says the user will like feature A, the product head thinks users will like feature B and so on and still other stalwarts think that actually feature C is what will bring everyone to their website. What happens ?

Scenario A:

They fight it out. Who wins? Everyone but the development team- all 3 features go in the sprint; developers somehow hack away at all 3 features and deliver something substandard. QA does not pass it or if they do they share the blame. 

Scenario B:

The technical project manager pushes back leaving a few people unhappy and the development team gets abused - "you guys always say no". 

Oh, there are so many ways this problem is attacked including a fight where the last man standing wins. 

But here is one interesting way we used on one of our projects. 

We used the much abused "Coming Soon" and see which feature generates the most interest from the users (clicks and comments). 

A naive implementation would  put up 3 different links and see which one gets the most clicks. However having 3 links leading to coming soon is not a great experience for users. So we showed just 1 coming soon link randomly to the users (much like A/B testing) and  we had our results. 

Again, there are so many ways you can do this but I prefer to use data to prioritize instead of egos. 

[eipico-blog]

Web Analytics - Getting Started

There is a plethora of tools for tracking online user behavior so if you get confused about which tools to use it is perfectly understandable. In fact if you are looking at web analytics tools for the first time and you are not confused then you haven't looked at enough tools:)

Here is my list of getting started tools:

Google Analytics (difficult to live online without this guy)    

 Register for Google Analytics

Set up exclusions for office IP address. Especially if you have a new website, this can mess around with your statistics. 

Set up basic goals that are triggered when a URL is visited. Initially this can be a thank you page when a user submits a “Contact Us” form which should be displayed on all pages on the website.

In later manifestations of your website Goal tracking is better implemented through event tracking.

HeapAnalytics (I prefer this over Clicky)

They capture all kinds of web analytics data. Their tag line is "capture everything". They start capturing all sorts of data as soon as you put the tag on your webpage (You are using Google Tag Manager, right?). And you can analyze this data through the tools and filters in HeapAnalytics. 

I suggest signing up for the "Free" plan for Startups. Up-to 25,000 recordings per month. If you have more visitors than that - good for you - pay up!

Clicky (Poor man's KissMetrics)

The basic plan is free so you can try it out, it is also useful when you have just started off and the number of visitors is small. 

This will help track the pages visited and interactions of individual users as opposed to aggregated information given my Google Analytics.

Register for a free version. This is free up to 3000 page visits/day.

Use the asynchronous tracking at the end of body tag without the brand image of Clicky.

In pro versions you can set up custom data so you can track the behavior of logged in users by username.

Mouseflow (Where does the mouse flow ...)

This tracks how the user moves their mouse over the page. 100 recordings a month are free; you can set the account to record, say 1 in 10 visits so that 100 recordings last for a longer time.

This is useful to determine how effective the UI is, whether the users get confused on where to click when they are on the page. 

[eipico-blog]

Digital Marketers - Stop Bothering the developers!

Maybe you are the guy in the company who has to run after developers to put that new cool plugin you found. Or, maybe, you are the star developer who has to halt everything to put in that new script and push it to production.

In either case Google Tag Manager is a godsend (or a "googlesend"!). Just bother your developers once more. Tell them you won't ask them to put in these scripts ever again and they will surprise you with their alacrity in putting in this script. 

Most of the web analytics tools are installed via JavaScript snippets on all pages of the website. Of course, you have a mechanism for doing that on your website. But, you can do better with Google Tag Manager. After the Google Tag Script is installed you can manage all the new scripts that you need to put in through this (with cool stuff like version control etc. )

Putting in google scripts is extremely straightforward(typically you just need the analytics ID or equivalent). And, putting in other scripts is also pretty simple. 

If you a Digital Marketer, get this script installed and see how their opinion about you shows a statistically significant improvement. 

Alternately, if you are a developer install this, trust your Digital Marketers to not screw things up (they are not actually as technologically challenged as you think they are!) and get them off your back. 

Then, please come back and thank me. 

[eipico-blog]

What's in a name?

The domain of this website deserves a little explanation in case it is not obvious. I have been fascinated by the elegance of this equation:

It is amazing how a transcendental number raised to the product of an imaginary number and another transcendental number yields a real integer.

You can read all about it here : http://en.wikipedia.org/wiki/Euler’s_identity

And this is a pretty good explanation of why e^(i *pi) + 1 = 0

One can ponder on this equation and wonder if this is just an accident determined by the way we chose to define mathematics or this is something ingrained in the structure of the universe itself.

But that would be opening a can of worm(hole)s; so let’s just pause here :)

But why did I choose this name? I can give you two explanations:

1.  In the spirit of this equation I intend to reduce the transcendental and the imaginary to something that is real. 

2. I like the name, and the domain was available. 

Now you can apply Occam's Razor ( among competing hypotheses the simpler one tends to be true) to these explanations and decide which one is more probable. I prefer the first one, of course.