A/B Testing with Fat Tails

59 Pages Posted: 12 May 2018 Last revised: 28 Feb 2019

See all articles by Eduardo M. Azevedo

Eduardo M. Azevedo

University of Pennsylvania - The Wharton School

Deng Alex

Microsoft Corporation

Jose Montiel Olea

New York University (NYU)

Justin M. Rao

Microsoft Research; Microsoft Corporation - Microsoft Research - Redmond

E. Glen Weyl

Plural Technology Collaboratory, Microsoft Research Special Projects; Plurality Institute; GETTING-Plurality Research Network

Date Written: February 26, 2019

Abstract

Large and thus statistically powerful A/B tests are increasingly popular in business and policy to evaluate potential innovations. We study how to optimally use scarce experimental resources to screen innovations. To do so, we propose a new framework for optimal experimentation that we call the A/B testing problem. The key insight of the model is that the optimal experimentation strategy depends on whether most gains accrue from typical innovations, or from rare and unpredictable large successes that can be detected using tests with small samples. We show that, if the tails of the (prior) distribution of true effect sizes is not too fat, the standard approach of trying a few high-powered experiments is optimal. However, when this distribution is very fat tailed, a lean experimentation strategy of trying more but smaller interventions is optimal. We measure this tail parameter using experiments from Microsoft Bing's EXP platform and find extremely fat tails. Our theoretical results and empirical analysis suggest that even simple changes to business practices within Bing could dramatically increase innovation productivity.

Keywords: A/B Testing, Innovation, Digitization, Randomized Trials, Bandit Problems, Optimal Experimentation

Suggested Citation

Azevedo, Eduardo M. and Alex, Deng and Montiel Olea, Jose and Rao, Justin M. and Weyl, Eric Glen, A/B Testing with Fat Tails (February 26, 2019). Available at SSRN: https://ssrn.com/abstract=3171224 or http://dx.doi.org/10.2139/ssrn.3171224

Eduardo M. Azevedo (Contact Author)

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

HOME PAGE: http://www.eduardoMazevedo.com

Deng Alex

Microsoft Corporation ( email )

One Microsoft Way
Redmond, WA 98052
United States

Jose Montiel Olea

New York University (NYU) ( email )

Bobst Library, E-resource Acquisitions
20 Cooper Square 3rd Floor
New York, NY 10003-711
United States

Justin M. Rao

Microsoft Research ( email )

641 Avenue of Americas
7th Floor
New York, NY 11249
United States

Microsoft Corporation - Microsoft Research - Redmond ( email )

Building 99
Redmond, WA
United States

Eric Glen Weyl

Plural Technology Collaboratory, Microsoft Research Special Projects ( email )

11 Ellsworth Ave, #2
Cambridge, MA 02139
United States
8579984513 (Phone)

HOME PAGE: http://www.glenweyl.com

Plurality Institute ( email )

GETTING-Plurality Research Network ( email )

124 Mount Auburn Street
Suite 520N
Cambridge, MA 02138
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
2,234
Abstract Views
8,725
Rank
12,536
PlumX Metrics