Large language models as research assistants: The risk of topic-overclassification in financial disclosure research

ABSTRACT Large language models (LLMs) are transforming unstructured text analysis. In accounting research, this raises measurement validity questions when disclosure-based constructs are derived from text. This study investigates LLMs as research assistants (RAs) for topic modeling and classification of financial disclosures. Comparing OpenAI’s GPT-4o with Latent Dirichlet Allocation (LDA) and human RAs, we evaluate LLM performance in identifying mergers and acquisitions (M&A) content in analyst reports. Results show document-specific LLM prompting generates more interpretable topics than LDA. However, GPT-4o exhibits high but lower reliability than human RAs. This difference reflects topic-overclassification risk: relative to consensus human benchmarks, GPT-4o applies a more inclusive classification profile, increasing the likelihood that contextually plausible but irrelevant text is classified as topic-relevant. Robustness tests using GPT-OSS 20B show that topic-overclassification depends on implementation choices and increases with additional contextual information. Additional analyses indicate that benchmark-relative positive classifications are concentrated in boilerplate-like and noisy text, while an ESG setting suggests that some disagreements with consensus human benchmarks may reflect human conservatism and ambiguity in construct boundaries. We conclude that apparent reliability can mask economically meaningful differences in classification profiles, requiring LLMs to be treated as measurement tools with transparent implementation choices, careful prompt design, and task-specific validation.