Bayesian Math versus Crowdsourcing
A Robust Bayesian Truth Serum for Small Populations http://www.eecs.harvard.edu/econcs/pubs/witkowski_aaai12.pdf
In this paper, we present the robust Bayesian Truth Serum
(RBTS) mechanism, which, to the best of our knowledge,
is the first peer prediction mechanism that does not rely on
knowledge of the common prior to provide strict incentive
compatibility for every number of agents n<3
. RBTS is also ex post individually rational (so that no agent makes a negative payment in any outcome) and numerically robust, being well defined for all possible agent reports.
A Bayesian Concept Learning Approach to Crowdsourcing http://www2.cs.uregina.ca/~zilles/viappianiZHB11idtgt.pdf
Quality Assessment for Crowdsourced Object Annotations http://cs.brown.edu/~hays/AnnotationQuality/AnnotationQuality.pdf
These results show that the number of control points
is indeed a good predictor of annotation quality except for the building category. We think that this metric is uniquely bad for the building category because many buildings are rectangular, thus many accurate annotations have only 4 control points.
A SIMULATION BASED ESTIMATION OF CROWD ABILITY AND ITS INFLUENCE ON CROWDSOURCED EVALUATION OF DESIGN CONCEPTS http://ode.engin.umich.edu/publications/PapalambrosPapers/2013/316.pdf
An important lesson from such community efforts is the need to implement a consistent method of filtering "signal" from "noise;" namely, obtaining valid contributions from those that are not. This general “signal to noise problem" manifests itself in any large crowd with heterogenous abilities, and in a crowd-sourced evaluation it is desirable to identify high ability participants from the rest of the crowd.