Dynamic Learning and Decision Making via Basis Weight Vectors

Abstract

This paper presents a new methodology to solve a general model of dynamic decision-making with a continuous unknown parameter or state. The methodology centers on the “continuation-value functions” (mappings from the parameter space to the continuation-value space), created by feasible continuation policies. When the model primitives can be described through a family of basis functions, e.g. polynomials, a continuation-value function retains that property and can be represented by a basis weight vector. The set of efficient basis weight vectors can be constructed through backward induction, which leads to a significant reduction of problem complexity and enables an exact solution for small-sized problems. A set of approximation methods based on the new methodology are developed to tackle larger problems. The methodology is also extended to the multi-dimensional (multi-parameter) setting, which features the problem of contextual multi-armed bandits with linear expected rewards. The approximation algorithm developed in this paper outperforms three benchmark algorithms (epsilon-greedy, Thompson Sampling, and LinUCB) in learning situations with many actions and short horizons.

Keywords: learning and doing, dynamic pricing with learning, linear contextual bandits, approximate dynamic programming, basis representation of functions

JEL Classification: C11, C44, C61, D83