Paying Attention to Inattentive Survey Respondents
Does attentiveness matter in survey responses? Do more attentive survey participants give higher quality responses? Using data from a recent online survey that identified inattentive respondents using instructed-response items, we demonstrate that ignoring attentiveness provides a biased portrait of the distribution of critical political attitudes and behavior. We show that this bias occurs in the context of both typical closed-ended questions and in list experiments. Inattentive respondents are common and are more prevalent among the young and less educated. Those who do not pass the trap questions interact with the survey instrument in distinctive ways: they take less time to respond; are more likely to report nonattitudes; and display lower consistency in their reported choices. Inattentiveness does not occur completely at random and failing to properly account for it may lead to inaccurate estimates of the prevalence of key political attitudes and behaviors, of both sensitive and more prosaic nature.
© 2019 The Author(s). Published by Cambridge University Press on behalf of the Society for Political Methodology. Published online: 16 January 2019. Levin thanks Sean Ingham for his collaboration in collecting the survey data used in this paper, and the University of Georgia for providing the financial support to conduct this survey. The survey data was collected using procedures approved by the Institute Review Board at the University of Georgia. Previous versions of this research were presented at the 34th Annual Meeting of the Society for Political Methodology (2017), at the 2017 Annual Meeting of the American Political Science Association (APSA), and at the 2nd Annual Southern California Methods Workshop at UCSB (September 19–20, 2017). We thank participants at these meetings for their comments and suggestions, and in particular we thank Michael James Ritter for his comments after our presentation at APSA, and Leah Stokes for her comments on our presentation at the Southern California Methods Workshop. Replication materials for this paper are available (Alvarez et al.2018).