Matthew Tyler - Model-Based Approaches to Resolving Human Coder Disagreement

Date
-
Location
Encina Hall West, Room 400 (GSL)
Speaker

Matthew Tyler, PhD Candidate, Stanford University

 

Abstract

Researchers within the social sciences, industry, and government are all tasked with placing objects into categories. For example, a large literature in comparative politics asks coders to evaluate a country’s level of democracy. Current best practice is to assign a small number of coders per case and then to resolve disagreement using a plurality vote of coders. In this project, I show that there are numerous advantages to using a probabilistic model instead. A model-based approach immediately yields a coherent mapping from coder decisions to the true category — all while making minimal assumptions about coder accuracy or agreement. For example, model-based predictions of the true label can learn to up-weight decisions from more accurate coders, while simple voting-based predictions cannot. I provide guidance on how to design coding tasks to get the most from probabilistic models, showing formally the benefits of increasing the number of coders assigned to each case. I apply these ideas to the measurement of democracy in thirteen countries, resolving disagreement among political science faculty in a way that objectively and automatically prioritizes subject-matter expertise.

 

Biography

Matt Tyler is a Ph.D. candidate with an interest in American politics and political methodology.