False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant

8 Pages Posted: 24 May 2011 Last revised: 29 Mar 2017

See all articles by Joseph P. Simmons

Joseph P. Simmons

University of Pennsylvania - The Wharton School

Leif D. Nelson

University of California, Berkeley - Haas School of Business

Uri Simonsohn

ESADE Business School; Ramon Llull University - ESADE Business School; ESADE Business School

Date Written: May 23, 2011

Abstract

In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

Keywords: Methodology, Motivated Reasoning, Publication, Disclosure

Suggested Citation

Simmons, Joseph P. and Nelson, Leif D. and Simonsohn, Uri, False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant (May 23, 2011). Psychological Science, 2011, Available at SSRN: https://ssrn.com/abstract=1850704

Joseph P. Simmons (Contact Author)

University of Pennsylvania - The Wharton School ( email )

3733 Spruce Street
Philadelphia, PA 19104-6374
United States

Leif D. Nelson

University of California, Berkeley - Haas School of Business ( email )

545 Student Services Building, #1900
2220 Piedmont Avenue
Berkeley, CA 94720
United States

Uri Simonsohn

ESADE Business School ( email )

Av. de Pedralbes, 60-62
Barcelona, 08034
Spain

Ramon Llull University - ESADE Business School ( email )

Avinguda de la Torre Blanca, 59
Sant Cugat del Vallès, 08172
Spain

HOME PAGE: http://urisohn.com

ESADE Business School ( email )

Av. de Pedralbes, 60-62
Barcelona, 08034
Spain

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
9,462
Abstract Views
48,005
Rank
1,111
PlumX Metrics