The Impact of Aberrant Responses and Detection in Forced‐Choice Noncognitive Assessment

Abstract The purpose of this study is to assess the impact of aberrant responses on the estimation accuracy in forced‐choice format assessments. To that end, a wide range of aberrant response behaviors (e.g., fake, random, or mechanical responses) affecting upward of 20%–30% of the responses was manipulated under the multi‐unidimensional pairwise preference (MUPP) model with forced‐choice format. Under each aberrance condition, we also computed the lz statistic (a common person‐fit index) to determine how well this index can accurately detect aberrant responses without erroneously misidentifying honest examinees. Some aberrant behaviors (e.g., limited responses) were more problematic than others (e.g., omitting) in terms of true trait level recovery. The lz statistic associated with each dimension of the MUPP form led to higher detection rates than did the overall lz statistic. When the dimensional lz approach was used, however, many honest examinees were misclassified as outliers. The detection rates of lz differed as a function of aberrant response types.