Approaches to Handling Common Response Styles and Issues in Educational Surveys

Survey has been a widely used data collection method for a variety of purposes in educational research. Although response styles have the potential to contaminate survey results, educational researchers often do little to control for such negative effects. Under discussion are five common response issues, their impact on survey data, and the methods that may be used to minimize the negative impact of these response issues on survey data. The five response issues in question are acquiescence (including disacquiescence), careless responding, extreme response, social desirability, and item-keying effect. Acquiescence (disacquiescence) refers to a respondent’s general tendency to agree (or disagree) with an item regardless of its content. This response style can distort item and construct correlations, compromising the results of factor analytic and correlational findings. Careless responding refers to a respondent’s tendency to pay insufficient attention to item content before responding, which can also lead to a biased estimation of relationships. Extreme response refers to the tendency of selecting extreme response options (e.g., strongly agree or strongly disagree) over middle options (e.g., neutral). Social desirability refers to a respondent’s tendency to rate him- or herself in an overly positive light. Finally, item-keying effect refers to a respondent’s differential responses to regular-keyed and reverse-keyed items. This effect often creates the illusion that items with opposite keying directions measure distinct constructs even when they may not. A growing amount of research has been done on how to control for the negative impact of these response styles, although the research may be limited and uneven for different response issues. A variety of approaches and methods exist for handling these response issues in research practice. Different response issues may require considerations at different stages of research. For example, effective handling of acquiescence response may require steps in both survey construction (e.g., including a hidden measure of acquiescence) and survey data analytic treatment (partial correlation technique), while controlling for item-keying effect may require more sophisticated modeling techniques (e.g., multitrait-multimethod confirmatory factor analysis).