The Use and Usefulness of Attention Checks in Behavioral Research

Abstract Attention checks have been touted as a means of excluding inattentive and careless responses from survey data. As a result, numerous approaches have been proposed, but which are most helpful has received less interest. The present paper demonstrates this blind spot has obscured vast differences in attention check efficacy. An initial survey of consumer-behavior and decision-making researchers revealed six commonly used attention check approaches (memory checks, questions with obvious answers, English comprehension, self-reported attentiveness, straight-lining, and variations on the Instructional Manipulation Check). These were then paired with six effects, drawn from the Many Labs series of papers in social psychology (N > 22,000, all pre-registered). The different checks varied substantially at reducing noise and increasing effect size. Follow-up studies suggest that participant attentiveness, as currently measured, may reflect a given participant’s general approach to completing studies, rather than momentary fatigue or boredom varying across time or within studies. We recommend behavioral researchers focus their efforts on going beyond mere “pulse-checkers” and consider favoring the use of memory checks and obvious questions as attention checks in their research.