How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice based on Over 60 Replicated Studies

260 Pages Posted: 16 Aug 2021 Last revised: 31 Mar 2023

See all articles by Apoorva Lal

Apoorva Lal

Stanford University

Mackenzie William Lockhart

affiliation not provided to SSRN

Yiqing Xu

Stanford University

Ziwen Zu

University of California, San Diego

Date Written: March 30, 2023

Abstract

Instrumental variable (IV) strategies are widely used in political science to establish causal relationships, but the identifying assumptions required by an IV design are demanding, and assessing their validity remains challenging. In this paper, we replicate 67 papers published in three top political science journals from 2010-2022 and identify several concerning patterns. First, researchers often overestimate the strength of their instruments due to non-i.i.d. error structures such as clustering. Second, the commonly used $t$-test for two-stage-least-squares (2SLS) estimates frequently underestimates uncertainty. Using more robust inferential methods, we find that about 19-30\% of the 2SLS estimates in our sample are underpowered. Third, in most replicated studies, 2SLS estimates are significantly larger than ordinary-least-squares estimates, with their ratio negatively correlated with instrument strength in studies with non-experimentally generated instruments, suggesting potential violations of unconfoundedness or exclusion restriction. We provide a checklist and software to help researchers avoid these pitfalls and improve their practice.

Keywords: instrumental variables, two-stage-least-squared, replications, weak instrument, exclusion restriction, replication

JEL Classification: C26

Suggested Citation

Lal, Apoorva and Lockhart, Mackenzie William and Xu, Yiqing and Zu, Ziwen, How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice based on Over 60 Replicated Studies (March 30, 2023). Available at SSRN: https://ssrn.com/abstract=3905329 or http://dx.doi.org/10.2139/ssrn.3905329

Apoorva Lal

Stanford University ( email )

Stanford, CA 94305
United States

HOME PAGE: http://apoorvalal.github,io

Mackenzie William Lockhart

affiliation not provided to SSRN

Yiqing Xu (Contact Author)

Stanford University ( email )

Stanford, CA 94305
United States

HOME PAGE: http://yiqingxu.org

Ziwen Zu

University of California, San Diego ( email )

CA
United States
3472442058 (Phone)
92092 (Fax)

HOME PAGE: http://www.ziwenzu.com

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
1,257
Abstract Views
3,784
Rank
30,262
PlumX Metrics