Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not'

FEDS Working Paper No. 2015-083

http://dx.doi.org/10.17016/FEDS.2015.083

26 Pages Posted: 6 Oct 2015

See all articles by Andrew C. Chang

Andrew C. Chang

Board of Governors of the Federal Reserve System

Phillip Li

Federal Deposit Insurance Corporation

Date Written: September 4, 2015

Abstract

We attempt to replicate 67 papers published in 13 well-regarded economics journals using author-provided replication files that include both data and code. Some journals in our sample require data and code replication files, and other journals do not require such files. Aside from 6 papers that use confidential data, we obtain data and code replication files for 29 of 35 papers (83%) that are required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that are not required to provide data and code replication files. We successfully replicate the key qualitative result of 22 of 67 papers (33%) without contacting the authors. Excluding the 6 papers that use confidential data and the 2 papers that use software we do not possess, we replicate 29 of 59 papers (49%) with assistance from the authors. Because we are able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is usually not replicable. We conclude with recommendations on improving replication of economics research.

Keywords: Data and Code Archives, Gross Domestic Product, Journals, Macroeconomics, National Income and Product Accounts, Replication

JEL Classification: B41, C80, C82, C87, C88, E01

Suggested Citation

Chang, Andrew C. and Li, Phillip, Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not' (September 4, 2015). FEDS Working Paper No. 2015-083, http://dx.doi.org/10.17016/FEDS.2015.083, Available at SSRN: https://ssrn.com/abstract=2669564 or http://dx.doi.org/10.2139/ssrn.2669564

Andrew C. Chang (Contact Author)

Board of Governors of the Federal Reserve System ( email )

20th Street and Constitution Avenue NW
Washington, DC 20551
United States

Phillip Li

Federal Deposit Insurance Corporation ( email )

550 17th Street NW
Washington, DC 20429
United States
202-898-3501 (Phone)
202-898-3500 (Fax)

HOME PAGE: http://https://www.fdic.gov/bank/analytical/cfr/bios/li.html

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
687
Abstract Views
3,551
Rank
69,812
PlumX Metrics