Profit-Driven Experimental Design
41 Pages Posted: 2 Aug 2021 Last revised: 22 Nov 2022
Date Written: July 30, 2021
Abstract
From intense competition to the recent pandemic, companies face considerable volatility in the business environment. For companies that design experiments to identify parameters of interest and make subsequent policy decisions based on these parameters, the cost of such experimentation has become increasingly comparable to the economic gains obtained, as the insights offered by an experiment can be short-lived due to changing market conditions. In this paper, we develop a general framework to quantify the total expected profit from both the experimental and postexperimental stages given an experimental strategy. The proposed framework is constructed using the asymptotic properties of the underlying parameter estimates as a channel to connect the profits from the two stages. We demonstrate that the order of the optimal sample size and its regret are critically shaped by the curvature of the postexperimental profit function, which is defined as a quantitative measure of the local sensitivity of the postexperimental profit function to the parameter of interest. By exploiting this property, we are able to identify the order of the optimal sample size and its regret without prior knowledge. We illustrate through demand-learning newsvendor and pricing problems that the curvature function, the order of the optimal sample size and its regret can be derived in closed-form, despite that the postexperimental profit function may not be fully specified. We also develop an algorithm to numerically compute the order of the optimal sample size when the curvature function cannot be solved in closed-form. Finally, when prior information is available, we provide the lower and upper bounds of the optimal sample size that maximizes the total expected profit.
Keywords: experimental design, asymptotic analysis, cost-effectiveness, sample size
JEL Classification: C90, C44
Suggested Citation: Suggested Citation