Contextual Learning with Online Convex Optimization: Theory and Application to Medical Decision-Making

Management Science, to appear.

63 Pages Posted: 31 Dec 2019 Last revised: 18 Nov 2023

See all articles by Esmaeil Keyvanshokooh

Esmaeil Keyvanshokooh

Mays Business School, Texas A&M University

Mohammad Zhalechian

Kelley School of Business, Indiana University

Cong Shi

Management Science, Herbert Business School, University of Miami

Mark P. Van Oyen

University of Michigan at Ann Arbor

Pooyan Kazemian

Department of Operations, Weatherhead School of Management, Case Western Reserve University, Cleveland, OH

Date Written: December 10, 2019

Abstract

Optimizing the treatment regimen is a fundamental medical decision-making problem. This can be thought of as a two-dimensional decision-making problem with a nested structure, because it involves determining both the optimal medication and its optimal dose. Identifying the most effective medication for an individual often poses considerable difficulty, and even when a suitable medication is ascertained, dosing it optimally remains a significant challenge. Making these two nested decisions necessitates the adaptive learning of a personalized disease progression control model. To address this problem, we propose a novel contextual multi-armed bandit model under a two-dimensional control with a nested structure. For this model, we develop a new joint contextual learning and optimization algorithm, termed the stochastic sub-gradient descent atop contextual bandit algorithm (SIENNA). It sequentially selects for a patient: (i) the best medication based on their contextual information, and (ii) the corresponding dose optimized over the prior history of those patients who received the same medication. We prove that it admits a sub-linear regret, which is tight up to a logarithmic factor. Our regret analysis leverages the strengths of both contextual bandit approaches and online convex optimization techniques in a seamless fashion. We substantiate the practicality of SIENNA using clinical data on patients with hypertension and heightened cardiovascular risks. Our analysis indicates that SIENNA has the potential to surpass current practices. We benchmark several policies to show the advantages of our approach and offer critical insights. Our framework holds promise for various applications beyond healthcare that require nested decision-making.

Keywords: online learning algorithms, regret analysis, contextual multi-armed bandit, stochastic sub-gradient descent, online convex optimization, personalized medicine, medical decision-making

Suggested Citation

Keyvanshokooh, Esmaeil and Zhalechian, Mohammad and Shi, Cong and Van Oyen, Mark P. and Kazemian, Pooyan, Contextual Learning with Online Convex Optimization: Theory and Application to Medical Decision-Making (December 10, 2019). Management Science, to appear., Available at SSRN: https://ssrn.com/abstract=3501316

Esmaeil Keyvanshokooh (Contact Author)

Mays Business School, Texas A&M University ( email )

430 Wehner
College Station, TX 77843
United States

HOME PAGE: http://ekshokooh.github.io/

Mohammad Zhalechian

Kelley School of Business, Indiana University ( email )

1275 E 10th St
Bloomington, IN 47405
United States

Cong Shi

Management Science, Herbert Business School, University of Miami ( email )

5250 University Dr
Coral Gables, FL 33146
United States

HOME PAGE: http://https://congshi-research.github.io/

Mark P. Van Oyen

University of Michigan at Ann Arbor ( email )

500 S. State Street
Ann Arbor, MI 48109
United States

Pooyan Kazemian

Department of Operations, Weatherhead School of Management, Case Western Reserve University, Cleveland, OH ( email )

10900 Euclid Ave.
Cleveland, OH 44106-7235
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
97
Abstract Views
2,334
Rank
492,371
PlumX Metrics