Beyond Overall Treatment Effects: Leveraging Covariates in Randomized Experiments Guided by Causal Structure

Tafti, Ali, and Galit Shmueli. "Beyond Overall Treatment Effects: Leveraging Covariates in Randomized Experiments Guided by Causal Structure." Information Systems Research 31.4 (2020): 1183-1199.

46 Pages Posted: 26 Feb 2019 Last revised: 29 Jan 2021

See all articles by Ali R. Tafti

Ali R. Tafti

University of Illinois at Chicago

Galit Shmueli

Institute of Service Science, National Tsing Hua University, Taiwan

Date Written: October 9, 2019

Abstract

Researchers using randomized controlled trials (RCTs) often subgroup or condition on auxiliary variables that are not the randomized treatment variable. There are many good reasons to condition on auxiliary variables—also referred to as control variables or covariates—in randomized experiments. In particular, designing and conducting RCTs is costly to researchers and subjects, and therefore it is important to derive greater value from RCT data; measuring not just the average treatment effect (ATE), but also finding more nuanced insights about the underlying theoretical mechanisms and generalizing the inferences. Unfortunately, there are many confusing and even contradictory guidelines on the use of subgroups or auxiliary variables in RCTs. We show how researchers can leverage covariates without biasing their causal inferences, by applying a few simple rules based on Judea Pearl’s causal diagramming framework. We demonstrate how to create a causal schema, through careful and deliberate operationalization of auxiliary covariates, in order to analyze the intermediate effects along a causal chain from the treatment to outcome; and we discuss some other ways to leverage covariates for theory development and generalization of findings from RCTs. We present a criterion for distinguishing pre-treatment and post-treatment variables that is based on directed acyclic graphs (DAGs). We provide a succinct set of guidelines to help readers begin to employ some essential techniques of DAG-based causal analysis. Finally, we provide a series of short tutorials (with accompanying simulated data and R scripts) to help readers explore the connections between RCT and observational contexts in causal diagramming. This commentary aims to raise awareness of the DAG methodology, explain its usefulness to experimental research, and encourage adoption in the IS community for studies using RCTs as well as observational data.

Keywords: randomized experiments, causal inference, randomized clinical trials, causal diagrams

Suggested Citation

Tafti, Ali R. and Shmueli, Galit, Beyond Overall Treatment Effects: Leveraging Covariates in Randomized Experiments Guided by Causal Structure (October 9, 2019). Tafti, Ali, and Galit Shmueli. "Beyond Overall Treatment Effects: Leveraging Covariates in Randomized Experiments Guided by Causal Structure." Information Systems Research 31.4 (2020): 1183-1199., Available at SSRN: https://ssrn.com/abstract=3331772 or http://dx.doi.org/10.2139/ssrn.3331772

Ali R. Tafti (Contact Author)

University of Illinois at Chicago ( email )

601 S Morgan St, 2403 University Hall, MC 294
Chicago, IL 60607
United States

Galit Shmueli

Institute of Service Science, National Tsing Hua University, Taiwan ( email )

Hsinchu, 30013
Taiwan

HOME PAGE: http://www.iss.nthu.edu.tw

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
521
Abstract Views
2,665
Rank
98,895
PlumX Metrics