Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance

Notices of the American Mathematical Society, 61(5), May 2014, pp.458-471

14 Pages Posted: 12 Aug 2013 Last revised: 5 Jul 2015

See all articles by David H. Bailey

David H. Bailey

Lawrence Berkeley National Laboratory; University of California, Davis

Jonathan Borwein

Royal Society of Canada; University of Newcastle (Australia); Australian Academy of Science (deceased)

Marcos Lopez de Prado

Cornell University - Operations Research & Industrial Engineering; Abu Dhabi Investment Authority; True Positive Technologies

Qiji Jim Zhu

Western Michigan University

Date Written: April 1, 2014

Abstract

We prove that high simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations, a practice we denote “backtest overfitting”. The higher the number of configurations tried, the greater is the probability that the backtest is overfit. Because most financial analysts and academics rarely report the number of configurations tried for a given backtest, investors cannot evaluate the degree of overfitting in most investment proposals.

The implication is that investors can be easily misled into allocating capital to strategies that appear to be mathematically sound and empirically supported by an outstanding backtest. Under memory effects, backtest overfitting leads to negative expected returns out-of-sample, rather than zero performance. This may be one of several reasons why so many quantitative funds appear to fail.

Keywords: backtest, historical simulation, probability of backtest over-fitting, investment strategy, optimization, Sharpe ratio, minimum backtest length, performance degradation

JEL Classification: G0, G1, G2, G15, G24, E44

Suggested Citation

Bailey, David H. and Borwein, Jonathan and Borwein, Jonathan and López de Prado, Marcos and López de Prado, Marcos and Zhu, Qiji Jim, Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance (April 1, 2014). Notices of the American Mathematical Society, 61(5), May 2014, pp.458-471, Available at SSRN: https://ssrn.com/abstract=2308659 or http://dx.doi.org/10.2139/ssrn.2308659

David H. Bailey

Lawrence Berkeley National Laboratory ( email )

1 Cyclotron Road
Berkeley, CA 94720
United States

HOME PAGE: http://www.davidhbailey.com

University of California, Davis ( email )

One Shields Avenue
Apt 153
Davis, CA 95616
United States

HOME PAGE: http://www.davidhbailey.com

Jonathan Borwein

Royal Society of Canada

University of Newcastle (Australia)

Australian Academy of Science (deceased)

Marcos López de Prado (Contact Author)

Abu Dhabi Investment Authority ( email )

211 Corniche Road
Abu Dhabi, Abu Dhabi PO Box3600
United Arab Emirates

HOME PAGE: http://www.adia.ae

Cornell University - Operations Research & Industrial Engineering ( email )

237 Rhodes Hall
Ithaca, NY 14853
United States

HOME PAGE: http://www.orie.cornell.edu

True Positive Technologies ( email )

NY
United States

HOME PAGE: http://www.truepositive.com

Qiji Jim Zhu

Western Michigan University ( email )

Kalamazoo, MI 49008
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
16,084
Abstract Views
66,840
Rank
443
PlumX Metrics