A Data Discourse

[timxdata]

Big Data - How much of it are you really going to use?

Amid all the excitement surrounding Big Data, I mentioned in an earlier post that what matters most is making the best use of your available data. While Big Data certainly has its advantages, it also tends to be very noisy and could present a data overload. How much of that Big Data are you really going to use? An interesting statistic from the article - "From an IBM study of more than 1,700 Chief Marketing Officers, 71% say they are under-prepared to deal with the 'data explosion' they face in the marketing arena, even as 79% say that customer analytics influence their strategy decisions." Read the full article here.

Also, check out Chartio.

[timxdata]

Nonprofits and Data

Shifting an organization into a data-driven culture that never had one to begin with, is a huge step. This is especially so for many nonprofits, where impacts might not be so easily measurable. Incremental steps can and should be taken to develop a data culture and metrics for measuring program outcomes. Deriving insights from data analysis can help to improve the effectiveness of an organization - it tells you what you're doing, what's working, what isn't, and how to improve.

An interesting statistic from the article - "A recent survey of about 400 nonprofits by the Nonprofit Technology Network found that while 89% of the nonprofits surveyed tracked data about their finances, only 50% consistently tracked data on how their work affected the people it was designed to help." The article also raises the point that many foundations are pushing nonprofits into the data-driven future by awarding money to organizations that have found ways to turn their lofty goals into specific data points measuring impact. While nonprofits still have some ways to go in this, the future is looking bright. The article is titled 'How Nonprofits Can Use Data to Solve the World's Problems' and can be read here.

[timxdata]

Twitter's Algorithms

While keeping myself abreast of tech news, I came across this very interesting, albeit technical, article on how Twitter uses real-time search to identify spiking search terms and tag them for relevance and purpose. It's essentially finding data on the move, and feeding that data into tailoring relevant advertising for both the client and consumer. Interestingly, Twitter's data scientists and engineers don't rely on lines of code for this purpose. Rather, they use a real-time computation system to identify search spikes, then let human beings all over the world annotate that data. As mentioned in the article, human annotators are able to tell Twitter's systems that searches for "Big Bird" aren't primarily referencing a children's show, but a political debate. This helps Twitter make trending topics smarter and more coherent. By The Verge, based here in NYC. Read the full article here.

[timxdata]

Visualize. Visualize. Visualize.

"If I can't picture it, I can't understand it."

Albert Einstein called it a long time ago.

Although the statistics and results of your data analysis are easy enough to represent in visualizations, the insights are a little more challenging.

Each visualization that you present (graphs, charts, diagrams) should always have one key, clear, concise, and easily digestible takeaway. Ideally, your visualization should represent that takeaway as simply as possible, retaining the core insight. I would always advise having visualizations aesthetically simple, with astutely-chosen colors for emphasis. For instance, sections that you want to highlight should be in brighter, bolder colors. If it's a pie chart, tone down the colors of other less important slices, or make them graduated. Insights should be a one-liner. Make it short, attention-grabbing, but accurate.

In December, along with an M.I.T. Economist, I presented the detailed results of a large evaluation project to the CEO and COO of Heifer International. Prior to spending 6 weeks in the Philippines collecting data through household surveys, we had carefully designed both our quantitative analysis methodology and instruments (surveys, interview material).

After analyzing the mass of data, I had to think about:

1. What key results should be visualized, and why?

2. What type of visualization would be most appropriate for the type of data I have?

3. Should I use the same type of visualization throughout the report, or mix it up?

4. Does that key takeaway shine through?

5. How would these visualizations fit into a presentation?

6. Are these visualizations web-ready?

All statistical analysis packages these days can churn out decent visualizations, but I think the above-mentioned points are still worth keeping at the forefront.

[timxdata]

Attributes of a Web Analyst Explored

Here's a great article I came across recently about 'What makes a great web analyst'.

The points laid out are highly relevant, but it should also be noted that these are attributes of a "perfect" web analyst (which realistically doesn't exist).

Avinash Kaushik (Web Analytics 2.0) makes a good point about hiring managers distilling down the core skills and to prioritize what they need, keeping in mind that ultimately if an analyst has a strong interest in the work he/she does, growth is bound to happen.

[timxdata]

Social Media Analytics Platforms

(Source: IBM)

Here's an article that trumpets Google Analytics as the #1 software for social media analytics, and then some.

IBM's BigSheets, which deals with Big Data, is also a really good tool for sentiment analysis, and what is increasingly called Social Data Analytics. Run it for 24 hours or a few days and see what you get!

[timxdata]

Trust Your Data

(source: GoodData)

Trust your data. As long as you’ve appropriated the right methodology, trust the results of your data analysis, even though it might not be what you had expected.

For instance, the microfinance development vehicle was thought and touted to be the world’s great solution for the rural poor all over the world. But extensive research, most notably by Esther Duflo (J-Pal at M.I.T.), suggested that microfinance really wasn’t making that great of an impact as was believed. Of course, the microfinance proponents were outraged. But the critical component here for Duflo’s claim was the result of rigorous data analysis using advanced econometric techniques.

We need to be willing to step back from our preconceived notions and accept what the data has to say, even if it's not what we want to hear. It is only in this way that we will really be able to internalize the virtues of data in telling us what is actually happening.

[timxdata]

Data-centric!

(source: WSJ)

An interesting article to kick off the new year from the WSJ blog, The Accelerators, where Warby Parker's Neil Blumenthal writes about developing data dashboards and pushing towards being a data-driven, data-centric business. He emphasizes, rightly so, for businesses to make use of the data they have, rather than thinking that Big Data is the only data.

Read the article here, and a complementary one here.

[timxdata]

Columbia Data Science

Rachel Schutt, a Senior Statistician at Google Research here in New York, started teaching an introductory course to Data Science at Columbia University's Statistics Department in the Fall of 2012.

Here is her blog for that class.

Very interesting material, with a lot of technical teaching about data science, and applying the correct methods to accurately and creatively solve problems using data analysis.

A thoroughly interesting interview with Rachel Schutt can also be found here.

[timxdata]

New Year's Hopes - Where I Think Data Analysis Should Be Headed

It's a new year, another 365 days to advance the frontier of data analysis.

This is where I think data analysis should be headed:

1. A big push towards ascertaining causality rather than just simple correlations. This means daring to incorporate (and actually doing so) marketing "experiments" into businesses and product development strategies.

2. THINK. Think about the data you're analyzing. Think about how you're analyzing it. Why are you doing it this way and not another? Think about what type of data you want to have, and design instruments to attain it. You don't have to be limited simply by what customers provide you. Actively seek out the data you want.

3. You don't necessarily need Big Data. From a data analysts' point of view, businesses should work with the data they have. And more importantly, make the best use out of it. It's surprising how I've seen so much useful 'small' data used for mundane insights, when there's so much businesses can extract from them.

4. Businesses tend still to be narrow-minded about the role of data analysts. If they're hiring a product analyst, they want someone who's experienced in product analysis. But that's already limiting the potential of what a data analyst can do. A data analysts' ability extends far beyond just one field. It is a skill that can be creatively applied to a whole myriad of tasks and fields. Be open-minded about maximizing a data analysts' experience from a variety of areas when hiring one.

5. As always, means should be presented with standard errors ;)

The field of data analysis is never static. It's constantly evolving, creating innovative approaches to use data. All too often, businesses focus on what you have done, rather than what you will be able to do. Let's hope 2013 is a year that dares to take leaps.

[timxdata]

Data Dashboards - You Need Them

(source: Google Analytics)

Data dashboards.

You need them. They come in all forms, from Web-based Google Analytics and Bing Webmaster Tools to software developed for your specific business.

There are many avenues to track your customers' preferences, behavioral trends, and continuing interest in your products - Twitter, Pinterest, Facebook, Svpply. Yet, you don't necessarily need Big Data if you're a small online start-up. It's how you use the data. Developing your data dashboard as you progress requires looking ahead. It also allows you to be more adept at predicting what your customers are likely to respond favorably to. What interesting correlations do I want to look at? I think this type of data might be useful - how do I incorporate this into the customer experience so that I can get the data I want?

An interesting NY Times article explores how small businesses use simple tools to sift data for eager customers, making it possible to tailor the online as well as offline experience to customers' specific preferences.

Main tenets of the article:

1. Capture Preferences

2. Respond Quickly

3. Listen to the Chatter

4. Build Relationships

The full article can be found here.

[timxdata]

Google Analytics in a Supermarket

Google Analytics recently envisioned how it would be like in real life for customers navigating a supermarket, but through a virtual interactive experience. Searching for items and purchasing them online is much quicker than going to a brick-and-mortar store, but that doesn't mean customers are any less patient. You'd be surprised at how much online customers value the ease of simplicity of purchasing what they want - and how quickly they leave the site (with no purchase) if it is virtually cumbersome.

This illustrates a point about how data analysis can look at the flow of traffic. From there, we can go even further to create an intentional roadmap that will allow customers to navigate as easily as possible to what they want to go, even if they take detours along the way.

View the full article here.

[timxdata]

Introduction

Data analytics is about solving problems, and coming up with new ideas. But it is also, more importantly, about how to make the best use of huge amounts of data wherein are contained information on human behavior and development. Being able to dig deep into this mass to find useful insights ultimately drives the whole process of data analysis. And it is immensely interesting.

While the task of analyzing huge amounts of data might be at first immediately daunting, there are several key things to make your process clear and systematic:

1. Know what type of data you're dealing with.

2. Determine what you ultimately want to get out of the data.

3. Determine what method is most appropriate for your type of data and the statistics you want to get out of it. 

4. Remember to always put the data into context when drawing insights.

These form a primer to taking a more scientific and nuanced investigation into data. There is a method in the "madness".

So here on this intellectual foray into data land, I'll do my best to enlighten and illuminate; make inroads into the heart of data analysis and what it means for tech, businesses, economics, and development.

This will be a multifaceted blog, delving as much into the science of data analysis as looking at its applications in the larger context, and where the frontier is advancing.