What’s the Makeup of a High Quality Estimize Analyst?
Crowdsourcing efforts strive to leverage user generated content into highly insightful data for new sales or to encourage additional contributions. Consider Wikipedia's business model, the classic crowdsourced tool for encyclopedia entries. If a majority of the pages contained errors or misinformed visitors, the site would no longer exist, but the community’s surprisingly self-healing capabilities help maintain quality control. Wikipedia now relies on millions of contributors to produce and edit almost 19 million different articles across more than 280 different languages, leading other organizations to explore the same framework in other sectors such as technology, business, and software programming.
However, not all businesses succeed at crowdsourcing. Attracting high quality users to act thoughtfully and for the greater good of building a better platform can fail amid internet trolls highjacking the process. Overcoming many of these individual limitations are integral to the success of Estimize over the last 5 years. The platform which currently consists of 50,000 estimates across 2,000 stocks faced similar obstacles found in other crowdsourcing efforts, namely finding a critical point of scale and quality.
Research on the Estimize data sets revealed that our growing list of participants act similarly to an editor on Wikipedia. A person’s willingness to contribute inherently originates from both intrinsic, a sense of internal validation that your estimate was accurate, and extrinsic motivations, a payoff or sense of community when Estimize sends you a trophy. Both motivating factors typically change depending on the user’s background and use case.
Finding the root of user’s behavior allows us to ascertain what makes an analyst more accurate relative to the general public or even a market expert. Many factors play a role in finding that sweet spot but none more than historical accuracy and revisions activity, ie: whether the user came back to edit an estimate after new information was available. While the best analysts are often the most active ones, our research concludes that sheer volume of estimates plays a minimal role in historical accuracy. Over time though, even the most precise users hit a rough patch in which accuracy reverts to the historical mean.
The importance of past performance and timeliness underpins the confidence score that users receive at the time they make an estimate on Estimize. Scores range from zero to 10, in which a low figure represents a subpar analyst on a given release or sector and a high score the exact opposite.
To calculate each score and thereby define an analyst’s accuracy, we first measure the time between an estimate and the actual print, normalized by the variance of other contributor’s error rates. In other words, an estimate the day before a report produces better results than those weeks or months in advance as it fully reflects all the available information to take place during the quarter. Using a similar approach on a sector level only sharpens the overall confidence score because analysts tend to specialize in one sector or segment of the market.
As for user’s background, the confidence score doesn’t factor in an analyst’s biographical background (i.e. buyside, sellside, independent trader, academic, etc.) to measure accuracy but a 2014 study from Deutsche Bank explored that very question. The report compared the accuracy of earnings forecasts from finance professionals, consisting of buy side, independent researchers and sell side analysts, with non professionals, a group of students, academics and sector experts. Consistent with the wisdom of crowds all estimates regardless of profession added to the efficacy of the Estimize data set, however a head to head comparison revealed that non professionals slightly edged out the experts. While the paper admitted the results were not statistically large enough to make any sweeping conclusions, it somewhat reaffirms any doubts cast on sell side research in recent years.
There exists an assortment of literature that sheds light on many of the biases influencing the sell side’s ratings and quarterly estimates. Many of them involve maintaining corporate access for clients and investment banking arms of firms. Issuing a scathing sell rating or revising financial estimates lower can cause a firm to lose key meetings with corporate executives and hence puts pressure on analysts to publish bullish reports on the publicly traded companies they track. It’s alarming that investors still adhere to analyst’s recommendations despite the clear favoritism and biases weighing on the industry.
Fortunately, a majority of this behavior is drowned out in the Estimize data set because it draws information from various sources beyond sell side analysts. And for that reason, many of the most accurate users on the platform hold non professional titles.
The path to becoming the best user varies between different crowdsourcing efforts, but one thing remains the same; persistence leads to better quality. For Estimize, that means estimating early and often for a given earnings release and throughout the sector. Our extensive research over the last 5 years reveals that this combination produces the best analysts and also builds a more accurate data set.
Want to Learn More? Visit Estimize or email us at [email protected]
Photo Credit: KamiPhuc
