Conducting Evaluations Using Multiple Trials

American Journal of Evaluation, Forthcoming

Posted: 8 Jul 2020

See all articles by David H. Greenberg

David H. Greenberg

University of Maryland Baltimore County

Burton S. Barnow

George Washington University - Trachtenberg School of Public Policy and Public Administration

Date Written: June 13, 2020

Abstract

This paper reviews the use of multiple trials in evaluating social programs. We define multiple trials to include both conducting evaluations in multiple sites concurrently and replications, where the intervention is replicated in one or more sites after the initial evaluation. After defining key terms, the paper discusses the rationales for conducting multiple trials, which include increasing sample size to increase statistical power; identifying the most effective program design; increasing external validity; and learning how various factors affect program impact. The paper discusses reasons why program design varies across sites, including adaptations to local environment and participant characteristics and a lack of fidelity to the program design. The issue of program fidelity, or uniformity, is complex, and the paper discusses why programs vary across sites and when it is desirable to maintain consistency and when variation should be permitted. Distinctions are drawn between evaluations of pilots and demonstrations versus ongoing programs, as well as programs where variation is permitted or encouraged versus when a fixed design is desired. The paper then discusses how multiple trials can be used to learn more about the implications of participant and place heterogeneity. The paper includes illustrations drawn from randomized controlled trial evaluations of demonstrations and ongoing programs and explains how decisions about the use of multiple trials should be built into initial evaluation designs and modified as results are obtained.

Keywords: multiple sites, replication, multiple arm evaluations, randomized control trials, program design

JEL Classification: J24, B41

Suggested Citation

Greenberg, David H. and Barnow, Burton S., Conducting Evaluations Using Multiple Trials (June 13, 2020). American Journal of Evaluation, Forthcoming, Available at SSRN: https://ssrn.com/abstract=3626018

David H. Greenberg

University of Maryland Baltimore County ( email )

1000 Hilltop Circle
Baltimore, MD 21250
United States
410-884-9620 (Phone)

Burton S. Barnow (Contact Author)

George Washington University - Trachtenberg School of Public Policy and Public Administration ( email )

805 21st Street, NW
Suite 601
Washington, DC 20052
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Abstract Views
182
PlumX Metrics