Being Honest in Backtest Reporting: A Template for Disclosing Multiple Tests

Abstract

Selection bias under multiple testing is a serious problem. From a practitioner’s perspective, failure to disclose the impact of multiple tests of a proposed investment strategy to clients and senior management can lead to the adoption of a false discovery. Clients will lose money, senior management will misallocate resources, and the firm may be exposed to reputational, legal and regulatory risks. From the perspective of academic journals that publish evidence supporting an investment strategy, the failure to address selection bias under multiple testing threatens to invalidate large portions of the literature in empirical finance. In this article, the authors propose a template that practitioners should use when pitching strategies to clients and senior management to fairly disclose multiple tests involved in an alleged discovery. The same template could be used by contributors to academic journals so that referees and ultimately readers can assess the strategy. By disclosing this information, those who are charged with making the final decision about a discovery can evaluate the probability that the purported discovery is false.

Keywords: selection bias, multiple testing, false positive, machine learning, clustering

JEL Classification: G0, G1, G2, G15, G24, E44