Yiqing Xu - How Much Should We Trust Estimates from Instrumental Variable Designs in Political Science? Practical Advice Based on Sixty-three Replications

Instrumental variable (IV) strategies are commonly used in political science to establish causal relationships, yet the identifying assumptions required by an IV design are demanding, and it remains challenging for researchers to correctly apply the method. We replicate 63 papers published in three top journals in political science during the past decade and find several troubling patterns: (1) researchers often miscalculate the first-stage F statistics, thus overestimating the strength of their IVs; (2) most researchers rely on classical asymptotic standard errors, which often severely underestimate the uncertainties around the two-stage-least-squared (2SLS) estimates; (3) in the majority of the replicated studies, the 2SLS estimates are much bigger than the ordinary-least-squared estimates, and their ratio is negatively correlated with the strength of the IVs when the IVs are non-experimental, suggesting potential violation of the exclusion restriction; such a relationship does not exist when the IVs are produced by experiments. To improve practice, we provide a checklist for researchers to avoid these pitfalls and recommend a local-to-zero test to guard against failure of the identifying assumptions.