Response Order Effects in Dichotomous Categorical Questions Presented Orally: The Impact of Question and Respondent Attributes

Abstract

Using data from 548 experiments in telephone surveys conducted by the Gallup Organization, we explored how attributes of questions and respondents moderate response order effects in dichotomous categorical questions. These effects were predominantly recency effects and occurred most in questions that were more difficult to comprehend (especially among respondents with the least education), with response choices that were more difficult to comprehend (because they were complete sentences instead of words or phrases and because they were not mutually exclusive), and that were asked after many prior questions. Recency effects were also more common in questions that explicitly or implicitly encouraged respondents to wait until they had heard all the answer choices before formulating a judgment than in questions that induced respondents to begin formulating a judgment before all the answer choices had been read (especially among the least educated respondents). A study of interviewer behavior revealed patterns of pausing between and within sentences that help to explain why some types of questions are especially prone to recency effects and others are not.