Generative AI as a Tool for Content Validation: Establishing Model-Specific Interpretive Benchmarks

Content validation is a core component of psychological scale development, traditionally conducted using human raters. Recent work suggests that generative artificial intelligence (AI) and large language models (LLMs) can serve as efficient surrogate raters when prompted with natural-language content validation tasks. However, evidence indicates that LLM-derived content validity estimates may systematically differ from human-based estimates, raising interpretive challenges. This study addresses this gap by developing model-specific interpretative benchmarks for LLM-based content validation. Using 112 published psychological scales from Colquitt et al. (2019), we evaluated content validity with three state-of-the-art lightweight LLMs (GPT-5 mini, Claude Sonnet 4, and Gemini 2.5 Flash) across two established methods: item sorting and item rating. Each scale was evaluated across ten independent iterations per model. Convergent validity was assessed by correlating LLM-derived content validity statistics with corresponding human-derived statistics. Results showed strong convergence between human and LLM estimates (rs = .605), with higher correlations for rating-based statistics than sorting-based statistics. Only rating-task statistics approached recommended convergence thresholds. LLM-derived estimates were consistently higher than human estimates, motivating the development of model-specific interpretative benchmarks. Percentile-based benchmarks are provided to guide the practical use of LLMs in scale content validation.