Your Attention Please! Toward a Better Understanding of Research Participant Carelessness

Applied PsychologyVolume 67, Issue 2 p. 227-230 Special IssueFree Access Your Attention Please! Toward a Better Understanding of Research Participant Carelessness Nathan A. Bowling, Corresponding Author Nathan A. Bowling nathan.bowling@wright.edu Wright State University, USAAddress for correspondence: Nathan A. Bowling, Department of Psychology, Wright State University, 3640 Colonel Glenn Highway, Dayton, Ohio 45435-0001, USA. Email: nathan.bowling@wright.eduSearch for more papers by this authorJason L. Huang, Jason L. Huang Michigan State University, USASearch for more papers by this author Nathan A. Bowling, Corresponding Author Nathan A. Bowling nathan.bowling@wright.edu Wright State University, USAAddress for correspondence: Nathan A. Bowling, Department of Psychology, Wright State University, 3640 Colonel Glenn Highway, Dayton, Ohio 45435-0001, USA. Email: nathan.bowling@wright.eduSearch for more papers by this authorJason L. Huang, Jason L. Huang Michigan State University, USASearch for more papers by this author First published: 31 January 2018 https://doi.org/10.1111/apps.12143Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Researchers have long been concerned that participants may respond carelessly to survey questionnaires (for early discussions of this problem, see Buechley & Ball, 1952; Greene, 1978; Haertzen & Hill, 1963). Some participants, for example, may hastily skim questionnaire items before responding; others may respond without reading the items at all (Huang, Curran, Keeney, Poposki, & DeShon, 2012; Meade & Craig, 2012). This presents a problem for researchers and practitioners, because the presence of even small amounts of careless responding1 can produce misleading research findings (see Credé, 2010; Huang, Liu, & Bowling, 2015; Schmitt & Stults, 1985). Concerns about careless responding have been further fuelled by the recent popularity of online questionnaires. Although a convenient data collection tool, online questionnaires may contribute to careless responding because they provide minimal researcher-participant social interaction and because they increase the potential for environmental distractions (see Meade & Craig, 2012). Against this backdrop, research on careless responding has flourished. Recent studies have examined several topics, including the measurement (e.g. Huang, Bowling, Liu, & Li, 2015; Maniaci & Rogge, 2014), prevention (e.g. Huang et al., 2012; Meade & Craig, 2012), causes (e.g. Bowling, Huang, Bragg, Khazon, Liu, & Blackmore, 2016; Gibson & Bowling, 2017), and consequences (e.g. Credé, 2010; Huang, Liu, & Bowling, 2015) of careless responding. Scientific progress in these areas has created new research possibilities and generated interesting questions about careless responding. The five articles included in the current special issue address several of these questions. Summary of Articles Included in the Current Special Issue Ward and Meade examine the effects of several experimental interventions on the presence of careless responding. Their interventions draw from established social psychological theories, specifically (a) social influence theories, (b) cognitive dissonance theory, and (c) social exchange theory. Across three datasets, Ward and Meade found mixed support for the effectiveness of their interventions. This paper offers insights into the motivational basis of careless responding and it provides a starting point for future efforts to develop interventions to prevent careless responding. Two papers included in the current issue examine whether embedding careless responding detection items affects participants' responses to substantive measures. Kung, Kwok, and Brown focus on instructed-response items and instructional manipulation check items. They reason that the inclusion of such items could cause participants to adopt a more deliberate mindset when responding to substantive measures, thus undermining the validity of their responses. Using two datasets, Kung et al. found that the inclusion of either instructed-response or instructional manipulation check items did not impact measurement properties of substantive measures, as indicated by scale means and tests of measurement invariance. As a result, they conclude that researchers can confidently use these items to assess careless responding without fear of undermining the validity of substantive measures. Similarly, Breitsohl and Steidelmüller assess potential influence of bogus items, instructed-response, or instructed manipulation check items. They argue that such attempts at detecting careless responding may affect responses to substantive measures because they can erode respondents' trust in researchers, cause them to feel insulted, or simply lead to more attentive responding. They also note that the presence of warning may provide justification for the detection methods and thus mitigate their potential negative impact. Using working adults recruited online from Austria, Germany, and Switzerland, they found that work-related substantive measures generally yielded invariant parameter estimates across different conditions. However, the reliability of substantive measures differed across conditions: presenting detection methods with warnings generally resulted in higher reliability estimates than presenting detection methods without warnings. DeSimone, DeSimone, Harms, and Wood use three simulated datasets to examine the effects of two types of careless responding—random responding and non-random responding ("straightlining"). This research is important because previous studies have not examined the differential effects of different types of careless responding (see Credé, 2010; Huang et al., 2015; McGonagle, Huang, & Walsh, 2016). DeSimone et al. found that random responding generally decreased the inter-item correlations, internal-consistency reliabilities, and first component eigenvalues of substantive measures. Non-random responding generally produced the opposite effects. Furthermore, the effects of non-random responding were generally more serious than those of random responding. Lovett, Bajaba, Lovett, and Simmering use a sample of Amazon's Mechanical Turk Masters (MTMs) to understand the attitudes and behaviours of experienced respondents on the popular crowdsourcing platform. They found that MTMs were primarily motivated to complete surveys as a means of earning money; as such, MTMs' perceptions of fairness in compensation not only influenced which surveys they chose to complete but could also affect their data quality. Furthermore, the majority of MTMs reported that they responded to surveys attentively without distractions. In sum, these findings provide insights into the quality of data collected via crowdsourcing. Future Research Directions Given the widespread use of questionnaire measures in applied psychology research and practice, progress in the understanding of careless responding may lead to improvements in the measurement of a variety of substantive variables, including employee personality traits, job attitudes, and work behaviours, to name just a few. Collectively, the articles included in the current special issue make important contributions to the existing careless responding literature. Despite recent progress, however, much is left to be learned about careless responding. We believe there are two areas where careless responding researchers should focus their future efforts. First, more attention should be given to the development of careless responding theory. Theory could help identify when, why, and how careless responding occurs—and what can be done to prevent it. Unfortunately, careless responding research to date has largely been atheoretical. Second, more attention should be given to the effective measurement of careless responding. Although considerable progress has been made in this area in recent years (see Huang et al., 2012; Maniaci & Rogge, 2014; Meade & Craig, 2012), careless responding measures should be further refined. And drawing from our previous suggestion that the field would benefit from theory development, efforts to examine the validity of careless responding indices should be guided by a thoroughly articulated nomological network of the careless responding construct. Previous validation efforts, however, have focused almost exclusively on the convergence between careless responding indices. Footnote 1 Researchers have used several terms to refer to this behaviour, including "careless responding" (Meade & Craig, 2012), "insufficient effort responding" (Huang et al., 2012), "participant inattention" (Maniaci & Rogge, 2014), and "random responding" (Schmitt & Stults, 1985). References Bowling, N.A., Huang, J.L., Bragg, C.B., Khazon, S., Liu, M., & Blackmore, C.E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218– 229. Buechley, R., & Ball, H. (1952). A new test of "validity" for the group MMPI. Journal of Consulting Psychology, 16, 299– 301. Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70, 596– 612. Gibson, A.M., & Bowling, N.A. (2017). Stop with the questions already! The effects of questionnaire length and incentives on careless responding. Manuscript submitted for publication. Greene, R.J. (1978). An empirically derived MMPI carelessness scale. Journal of Clinical Psychology, 34, 407– 410. Haertzen, C.A., & Hill, H.E. (1963). Assessing subjective effects of drugs: an index of carelessness and confusion for use with the Addiction Research Center Inventory (ARCI). Journal of Clinical Psychology, 19, 407– 412. Huang, J.L., Bowling, N.A., Liu, M., & Li, Y. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299– 311. Huang, J.L., Curran, P.G., Keeney, J., Poposki, E.M., & DeShon, R.P. (2012). Detecting and deterring insufficient effort respond to surveys. Journal of Business and Psychology, 27, 99– 114. Huang, J.L., Liu, M., & Bowling, N.A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828– 845. Maniaci, M.R., & Rogge, R.D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61– 83. McGonagle, A.K., Huang, J.L., & Walsh, B.M. (2016). Insufficient effort survey responding: An under-appreciated problem in work and organizational health psychology research. Applied Psychology: An International Review, 65, 287– 321. Meade, A.W., & Craig, S.B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437– 455. Schmitt, N., & Stults, D.M. (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9, 367– 373. Citing Literature Volume67, Issue2Special Issue: Research Participant CarelessnessApril 2018Pages 227-230 ReferencesRelatedInformation