Designing Incentives for Inexpert Human Raters

11 Pages Posted: 25 Jun 2011 Last revised: 27 May 2017

See all articles by Aaron Shaw

Aaron Shaw

Harvard University - Berkman Klein Center for Internet & Society; University of California, Berkeley - Department of Sociology

John J. Horton

New York University (NYU) - Department of Information, Operations, and Management Sciences

Daniel L. Chen

Directeur de Recherche, Centre National de la Recherche Scientifique, Toulouse School of Economics, Institute for Advanced Study in Toulouse, University of Toulouse Capitole, Toulouse, France

Date Written: March 1, 2011

Abstract

The emergence of online labor markets makes it far easier to use individual human raters to evaluate materials for data collection and analysis in the social sciences. In this paper, we report the results of an experiment – conducted in an online labor market – that measured the effectiveness of a collection of social and financial incentive schemes for motivating workers to conduct a qualitative, content analysis task. Overall, workers performed better than chance, but results varied considerably depending on task difficulty. We find that treatment conditions which asked workers to prospectively think about the responses of their peers – when combined with financial incentives – produced more accurate performance. Other treatments generally had weak effects on quality. Workers in India performed significantly worse than US workers, regardless of treatment group.

Note: Most heavily-cited paper from the 2011 ACM-CSCW proceedings (as of November 27th, 2013), according to Google Scholar Best Paper Award Honorable Mention.

Keywords: Experimentation, Amazon Mechanical Turk, Human Computation, Crowdsourcing, Search, Content Analysis, Economics, Sociology, Experimentation, Measurement, Human Factors

Suggested Citation

Shaw, Aaron and Horton, John J. and Chen, Daniel L., Designing Incentives for Inexpert Human Raters (March 1, 2011). Proceedings of the Association for Computing Machinery Conference on Computer Supported Cooperative Work, 2011, Available at SSRN: https://ssrn.com/abstract=1871330

Aaron Shaw (Contact Author)

Harvard University - Berkman Klein Center for Internet & Society ( email )

Harvard Law School
23 Everett, 2nd Floor
Cambridge, MA 02138
United States

University of California, Berkeley - Department of Sociology

410 Barrows Hall
Berkeley, CA 94720
United States

John J. Horton

New York University (NYU) - Department of Information, Operations, and Management Sciences ( email )

44 West Fourth Street
New York, NY 10012
United States
6175952437 (Phone)

HOME PAGE: http://john-joseph-horton.com

Daniel L. Chen

Directeur de Recherche, Centre National de la Recherche Scientifique, Toulouse School of Economics, Institute for Advanced Study in Toulouse, University of Toulouse Capitole, Toulouse, France ( email )

Toulouse School of Economics
1, Esplanade de l'Université
Toulouse, 31080
France

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
82
Abstract Views
1,685
Rank
543,429
PlumX Metrics