Designing Incentives for Inexpert Human Raters
11 Pages Posted: 25 Jun 2011 Last revised: 27 May 2017
Date Written: March 1, 2011
Abstract
The emergence of online labor markets makes it far easier to use individual human raters to evaluate materials for data collection and analysis in the social sciences. In this paper, we report the results of an experiment – conducted in an online labor market – that measured the effectiveness of a collection of social and financial incentive schemes for motivating workers to conduct a qualitative, content analysis task. Overall, workers performed better than chance, but results varied considerably depending on task difficulty. We find that treatment conditions which asked workers to prospectively think about the responses of their peers – when combined with financial incentives – produced more accurate performance. Other treatments generally had weak effects on quality. Workers in India performed significantly worse than US workers, regardless of treatment group.
Note: Most heavily-cited paper from the 2011 ACM-CSCW proceedings (as of November 27th, 2013), according to Google Scholar Best Paper Award Honorable Mention.
Keywords: Experimentation, Amazon Mechanical Turk, Human Computation, Crowdsourcing, Search, Content Analysis, Economics, Sociology, Experimentation, Measurement, Human Factors
Suggested Citation: Suggested Citation
Do you have negative results from your research you’d like to share?
Recommended Papers
-
Running Experiments on Amazon Mechanical Turk
By Gabriele Paolacci, Jesse Chandler, ...
-
The Online Laboratory: Conducting Experiments in a Real Labor Market
By John J. Horton, David G. Rand, ...
-
The Online Laboratory: Conducting Experiments in a Real Labor Market
By John J. Horton, David G. Rand, ...
-
Conducting Behavioral Research on Amazon's Mechanical Turk
By Winter Mason and Siddharth Suri
-
The Labor Economics of Paid Crowdsourcing
By John J. Horton and Lydia B. Chilton
-
On the Conjunction Fallacy in Probability Judgment: New Experimental Evidence
By Gary Charness, Edi Karni, ...
-
Economic Games on the Internet: The Effect of $1 Stakes
By Ofra Amir, David G. Rand, ...