This study examines whether large language models (LLMs) can extract value-relevant information from narrative disclosures in earnings announcements that explains stock market reactions beyond traditional financial measures. The study focuses on CEO commentary sections of Q2 and Q4 earnings announcements published by 23 Finnish listed companies on the Nasdaq Helsinki exchange over the period 2014–2024. Narrative tone is measured using GPT-4 in a zero-shot setting and analysed together with analyst-based earnings surprise measures and firm-level control variables within a short-window event study framework. Cumulative abnormal returns over a three-day event window are used as the dependent variable. Three regression specifications are estimated: a baseline model including earnings surprises and controls, a full model that adds the LLM-based tone measure, and a moderation model that examines whether narrative tone influences the market’s response to earnings surprises. The results show that both revenue and EBIT surprises are positively related to announcement-period returns, consistent with the earnings response coefficient framework. The LLM-based tone measure is also positive and statistically significant, indicating that more optimistic CEO commentary is associated with higher abnormal returns. Including narrative tone increases the explanatory power of the model, supporting the view that qualitative disclosures provide information beyond financial figures. The moderation analysis shows that narrative tone weakens the market’s response to revenue surprises, suggesting that narrative disclosures influence how financial information is interpreted by investors. Robustness checks using alternative event windows and a continuous tone measure confirm the stability of the findings.