Is human decision-making dead in the era of Big Data?
I was impressed by a baseball movie staring Brad Pitt named Money Ball, which talks about statistical analysis in Baseball. With the data oriented philosophy, Billy Bean, the general manger of Oakland A’s achieves great success. A good question has been brought out in the film: which is more reliable? Computer algorithms or scouts’ eyes.
In the past, baseball players can only be judged by experienced scouts, who always make vague and unfalsifiable comment like “He’s a real player”. But now, with the great impact of data crunching, baseball managers are able to make more rational and unbiased decisions. For example, a player gets on base well but looks fat might be ignored by scouts, but data never tells a lie. Generally speaking, numbers rule out all factors might lead to biases, like personal preferences and emotions.
However, does it really mean human decision-making is dead? I don’t think so.
Some people predict that scout is a job which will be phased out in the future. It might be. But I still think there’s something cannot totally be interpreted by data analysis. The mental, psychological things can only be detected through personal observations and interactions. George Steinbrenner, the former president of New York Yankees, once told people that he’s been around for years and seen so many players, and he could really tell the difference by first sight. When he saw in to Derek Jeter’s eyes, he really saw a champion player who’s determined and confident. Mental things are always too intangible to put in a regression model.
Even though data crunching is the new smart today, I still insist that there’s room for human decision-making. Super Crunchers, the book I recently read, makes a good point about this. Even experts fail to outperform computers in terms of predictions and estimations in most cases, we still need them to determine what factors should be taken into account, what variables should be put into the equations. Data itself will never be able to do this.