Stability of Experimental Results: Forecasts and Evidence

62 Pages Posted: 21 May 2019

See all articles by Stefano DellaVigna

Stefano DellaVigna

University of California, Berkeley; National Bureau of Economic Research (NBER)

Devin G. Pope

University of Chicago - Booth School of Business

Multiple version iconThere are 2 versions of this paper

Date Written: May 2019

Abstract

How robust are experimental results to changes in design? And can researchers anticipate which changes matter most? We consider a specific context, a real-effort task with multiple behavioral treatments, and examine the stability along six dimensions: (i) pure replication; (ii) demographics; (iii) geography and culture; (iv) the task; (v) the output measure; (vi) the presence of a consent form. We use rank-order correlation across the treatments as measure of stability, and compare the observed correlation to the one under a benchmark of full stability (which allows for noise), and to expert forecasts. The academic experts expect that the pure replication will be close to perfect, that the results will differ sizably across demographic groups (age/gender/education), and that changes to the task and output will make a further impact. We find near perfect replication of the experimental results, and full stability of the results across demographics, significantly higher than the experts expected. The results are quite different across task and output change, mostly because the task change adds noise to the findings. The results are also stable to the lack of consent. Overall, the full stability benchmark is an excellent predictor of the observed stability, while expert forecasts are not that informative. This suggests that researchers' predictions about external validity may not be as informative as they expect. We discuss the implications of both the methods and the results for conceptual replication.

JEL Classification: C9, C91, C93

Suggested Citation

DellaVigna, Stefano and Pope, Devin G., Stability of Experimental Results: Forecasts and Evidence (May 2019). University of Chicago, Becker Friedman Institute for Economics Working Paper No. 2019-76, Available at SSRN: https://ssrn.com/abstract=3391520 or http://dx.doi.org/10.2139/ssrn.3391520

Stefano DellaVigna

University of California, Berkeley ( email )

Economics Department
549 Evans Hall #3880
Berkeley, CA 94720
United States
510-643-0715 (Phone)
510-642-6615 (Fax)

HOME PAGE: http://emlab.berkeley.edu/users/sdellavi/

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Devin G. Pope (Contact Author)

University of Chicago - Booth School of Business ( email )

5807 S. Woodlawn Avenue
Chicago, IL 60637
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
67
Abstract Views
658
Rank
479,249
PlumX Metrics