Policy Evaluation, Randomized Controlled Trials, and External Validity – A Systematic Review

51 Pages Posted: 2 Feb 2016

See all articles by Jörg Ankel-Peters

Jörg Ankel-Peters

RWI - Leibniz Institute for Economic Research; University of Passau - Business Administration and Economics

Joerg Langbein

Rhine-Westphalia Institute for Economic Research (RWI-Essen)

Gareth Roberts

University of the Witwatersrand

Date Written: January 25, 2016

Abstract

When properly implemented, Randomized Controlled Trials (RCT) can achieve a high degree of internal validity. Yet, if an RCT is to inform policy interventions that extend beyond the experimental population, it is critical to establish external validity. In this paper, we first present a theoretical framework of external validity and identify the potential hazards that compromise generalizing results beyond the studied population, namely Hawthorne effects, general equilibrium effects, specific sample problems, and special care in the provision of the randomized treatment. Second, we reviewed all RCTs published in leading economic journals between 2009 and 2014 and scrutinized the way they deal with external validity. Based on a set of objective indicators, we find that many published RCTs do not discuss hazards to external validity and do not provide the information that is necessary to assess potential problems. Apparently, external validity is not an important matter of concern during the peer review process. To conclude, we call for a more systematic approach to report the results of RCTs, including external validity dimensions.

Keywords: Systematic review; internal validity; external validity; randomized controlled trials

JEL Classification: C83; C93

Suggested Citation

Ankel-Peters, Jörg and Langbein, Joerg and Roberts, Gareth, Policy Evaluation, Randomized Controlled Trials, and External Validity – A Systematic Review (January 25, 2016). Ruhr Economic Paper No. 589, Available at SSRN: https://ssrn.com/abstract=2721679 or http://dx.doi.org/10.2139/ssrn.2721679

Jörg Ankel-Peters (Contact Author)

RWI - Leibniz Institute for Economic Research ( email )

Hohenzollernstr. 1-3
45128 Essen
Germany
0049-201-8149-247 (Phone)

HOME PAGE: http://www.rwi-essen.de

University of Passau - Business Administration and Economics ( email )

Wirtschaftswissenschaftliche Fakultaet
Passau, D-94030
Germany

Joerg Langbein

Rhine-Westphalia Institute for Economic Research (RWI-Essen) ( email )

Hohenzollernstr. 1-3
Essen, 45128
Germany

Gareth Roberts

University of the Witwatersrand ( email )

1 Jan Smuts Avenue
Johannesburg, GA Gauteng 2000
South Africa

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
186
Abstract Views
988
Rank
296,243
PlumX Metrics