The Probability of Backtest Overfitting

Journal of Computational Finance (Risk Journals), 2015, Forthcoming

34 Pages Posted: 16 Sep 2013 Last revised: 5 Jul 2015

See all articles by David H. Bailey

David H. Bailey

Lawrence Berkeley National Laboratory; University of California, Davis

Jonathan Borwein

Royal Society of Canada; University of Newcastle (Australia); Australian Academy of Science (deceased)

Marcos Lopez de Prado

Cornell University - Operations Research & Industrial Engineering; Abu Dhabi Investment Authority; True Positive Technologies

Qiji Jim Zhu

Western Michigan University

Multiple version iconThere are 2 versions of this paper

Date Written: February 27, 2015

Abstract

Most firms and portfolio managers rely on backtests (or historical simulations of performance) to select investment strategies and allocate them capital. Standard statistical techniques designed to prevent regression over-fitting, such as hold-out, tend to be unreliable and inaccurate in the context of investment backtests. We propose a framework that estimates the probability of backtest over-fitting (PBO) specifically in the context of investment simulations, through a numerical method that we call combinatorially symmetric cross-validation (CSCV). We show that CSCV produces accurate estimates of the probability that a particular backtest is over-fit.

The appendices for this paper are available at the following URL: http://ssrn.com/abstract=2568435

Keywords: backtest, historical simulation, probability of backtest over-fitting, investment strategy, optimization, Sharpe ratio, minimum backtest length, performance degradation

JEL Classification: G0, G1, G2, G15, G24, E44

Suggested Citation

Bailey, David H. and Borwein, Jonathan and Borwein, Jonathan and López de Prado, Marcos and López de Prado, Marcos and Zhu, Qiji Jim, The Probability of Backtest Overfitting (February 27, 2015). Journal of Computational Finance (Risk Journals), 2015, Forthcoming, Available at SSRN: https://ssrn.com/abstract=2326253 or http://dx.doi.org/10.2139/ssrn.2326253

David H. Bailey

Lawrence Berkeley National Laboratory ( email )

1 Cyclotron Road
Berkeley, CA 94720
United States

HOME PAGE: http://www.davidhbailey.com

University of California, Davis ( email )

One Shields Avenue
Apt 153
Davis, CA 95616
United States

HOME PAGE: http://www.davidhbailey.com

Jonathan Borwein

Royal Society of Canada

University of Newcastle (Australia)

Australian Academy of Science (deceased)

Marcos López de Prado (Contact Author)

Abu Dhabi Investment Authority ( email )

211 Corniche Road
Abu Dhabi, Abu Dhabi PO Box3600
United Arab Emirates

HOME PAGE: http://www.adia.ae

Cornell University - Operations Research & Industrial Engineering ( email )

237 Rhodes Hall
Ithaca, NY 14853
United States

HOME PAGE: http://www.orie.cornell.edu

True Positive Technologies ( email )

NY
United States

HOME PAGE: http://www.truepositive.com

Qiji Jim Zhu

Western Michigan University ( email )

Kalamazoo, MI 49008
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
16,045
Abstract Views
57,586
Rank
447
PlumX Metrics