The Sensitivity of Experimental Impact Estimates: Evidence from the National Jtpa Study
50 Pages Posted: 18 Mar 1998 Last revised: 9 Mar 2022
Date Written: July 1997
Abstract
The recent experimental evaluation of the U.S. Job Training Partnership Act (JTPA) program found negative effects of training on the earnings of disadvantaged male youth and no effect on the earnings of disadvantaged female youth. These findings provided justification for Congress to cut the budget of JTPA's youth component by over 80 percent. In this paper, we examine the sensitivity of the experimental impact estimates along several dimensions of construction and interpretation. We find that the statistical significance of the male youth estimates is extremely fragile and that the magnitudes of the estimates for both youth groups are sensitive to nearly all the factors we consider. In particular, accounting for experimental control group members who substitute training from other providers leads to a much more positive picture regarding the effectiveness of JTPA classroom training. Our study indicates the value of sensitivity analyses in experimental evaluations and illustrates that experimental impact estimates, like those from nonexperimental analyses, require careful interpretation if they are to provide a reliable guide to policymakers.
Suggested Citation: Suggested Citation
Do you have negative results from your research you’d like to share?
Recommended Papers
-
Characterizing Selection Bias Using Experimental Data
By James J. Heckman, Hidehiko Ichimura, ...
-
Propensity Score Matching Methods for Non-Experimental Causal Studies
By Rajeev H. Dehejia and Sadek Wahba
-
Propensity Score Matching Methods for Non-Experimental Causal Studies
By Rajeev H. Dehejia and Sadek Wahba
-
Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs
By Orley Ashenfelter and David Card
-
Causal Effects in Non-Experimental Studies: Re-Evaluating the Evaluation of Training Programs
By Rajeev H. Dehejia and Sadek Wahba
-
Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review
-
The Role of the Propensity Score in Estimating Dose-Response Functions
-
Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?
By Jeffrey A. Smith and Petra Todd