Using Focus Groups and Rasch Item Response Theory to Improve Instrument Development

Item response theory (IRT) is not widely used in counseling psychology research, despite its considerable advantages for instrument development. Focus groups help create a broad and representative item pool that is more likely to tap the full range of the latent dimension, thereby capitalizing on IRT strengths. We provide suggestions for using these tools, with an empirical example, the Everyday Multicultural Competencies/Revised Scale of Ethnocultural Empathy (EMC/RSEE). Rasch IRT methods were used to (a) analyze response format performance and collapse poorly performing categories, (b) evaluate differential item functioning sex bias, and (c) select items to maximize subscale sensitivity and bandwidth. Traditional classical test theory (CTT) subscales composed of items with the highest factor loadings were compared with subscales based on IRT criteria. Compared with CTT subscales, IRT subscales used in the EMC/RSEE demonstrated generally higher correlations with other variables of interest, and superior sensitivity to change over time and to group differences.