Improving the Statistical Power and Reliability of Research Using Amazon Mechanical Turk

41 Pages Posted: 28 Feb 2017 Last revised: 11 Jan 2021

See all articles by Jeremiah W. Bentley

Jeremiah W. Bentley

Isenberg School of Management, University of Massachusetts Amherst

Date Written: December 29, 2020

Abstract

Amazon Mechanical Turk (MTurk) is an increasingly popular source of experimental subjects for accounting research due to its convenience and low cost (relative to traditional laboratories). However, MTurk presents challenges related to statistical power and reliability. These challenges are not unique to MTurk, but are more prevalent than in research conducted with other participant pools. In this paper I discuss several reasons why research conducted with MTurk may face additional power and reliability challenges. I then present suggestions for dealing with these challenges, taking advantage of the comparative strengths of MTurk. The discussion should be of interest to PhD students and other researchers considering using MTurk or other online platforms as a source of experimental participants as well as to reviewers and editors who are considering quality control standards for research conducted with this participant pool.

Keywords: Amazon Mechanical Turk; MTurk; AMT; Effect Size; Sample Size; Screening

JEL Classification: M40, M41, M42, C18, C90, C91

Suggested Citation

Bentley, Jeremiah W., Improving the Statistical Power and Reliability of Research Using Amazon Mechanical Turk (December 29, 2020). Available at SSRN: https://ssrn.com/abstract=2924876 or http://dx.doi.org/10.2139/ssrn.2924876

Jeremiah W. Bentley (Contact Author)

Isenberg School of Management, University of Massachusetts Amherst ( email )

Department of Accounting
Amherst, MA 01003
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
705
Abstract Views
3,042
Rank
67,557
PlumX Metrics