How Does Estimize Maintain Data Integrity?
In part 2 of this series we look at why contributors consistently publish high quality and accurate data.
If you missed part 1, click here to catch up!
The principle of reciprocity (give to get) behind the Estimize system provides an equal or greater reward for a single estimate. For that reason, users operate under the belief that to receive an honest estimate they must also contribute one. That mentality alone does not protect the integrity of the data and it takes other measures to ensure participant’s behaviors add value instead of detract it.
By creating a simple scoring system, users are forced to publish both accurate and representative estimates. We score analysts on each earnings report for the level of accuracy relative to the consensus and their peers. One way users can earn a higher score is to publish aggressive estimates relative to the distribution, meaning an estimate that’s 100% accurate won’t score the maximum amount of points. We display scores directly on the platform and email results immediately after a report to discourage inaccurate estimating practices
Behavioral psychology contends that users feel a greater sense of failure when they fall behind their peers. Technically an analyst could contribute to Estimize buy putting their estimate simply 1 cent above EPS consensus and 1 dollar above revenue consensus, but that method runs into a roadblock. Our platform continually shames users making this mistake into inputting a real or honest estimate by giving them a lower confidence score. Do this enough times and users eventually start to provide real estimates without our poking and prodding.
At this point we don’t see users going down this rabbit hole as often because it takes less time and effort for to provide a real answer early and often. At one point we were concerned that this behavior would taint the data set, but after countless iterations and training users to estimate honestly, contribution rates and accuracy are nearing a tipping point.
In the opposite vein, some users feel the need to make outlandish estimates. To protect the data from those threats, we use a reliability algorithm that excludes certain estimates in the calculation of the consensus. The platform automatically flags outliers, we then manually review them within 30 minutes of the estimate being made before a decision is made to leave or remove the flag.
We also developed a multivariate regression model which uses a few dozen variables to weigh each estimate in the Estimize Consensus. They include the recency of the estimate, historical accuracy of an analyst in a given sector, the user’s structured biographical background submitted to the site, number of estimates, length of history, and certain behaviors taken on the site before making estimates (time on page, what they touch, etc.). These factors allow us to intelligently weigh the consensus in favor of the trustworthy analysts who display behavior consistent with accuracy.
Our philosophy unquestionably departs from the traditional course that other financial companies take to collect data but we remain confident that our superior methodology will inevitably subsume Wall Street to become the de facto consensus number on the market.
Photo Credit: Joel Tonyan