Emotion Word Processing in Mandarin-Speaking Adults With Hearing Aids and Cochlear Implants: The Interplay of Semantics, Prosody, and Facial Cues

Objectives: The processing of emotional speech requires the perception and integration of semantics, prosody, and facial cues. While hearing aids (HAs) and cochlear implants (CIs) provide significant auditory enhancements, they are limited in conveying emotional prosody, potentially affecting users’ ability to recognize emotions through both auditory and visual modalities. This study aims to compare the performance of HA users, CI users, and individuals with normal hearing in decoding emotions from various channels and to assess their channel preferences. Design: Using a forced-choice audiovisual Stroop paradigm, we recruited 23 HA users, 18 CI users, and 21 individuals with normal hearing to complete 3 channel-specific tasks: a semantics task, a prosody task, and a face task. Each task was designed to extract emotion-related information from a specific channel in accordance with the task’s requirements. Results: The prosody-judgment abilities of both hearing-impaired groups (CI users and HA users) were significantly impaired compared with individuals with normal hearing. In addition, during multimodal processing, the CI group showed no statistically detectable cross-channel interference on the face task, whereas in the HA group, semantic inconsistency was associated with lower performance across all three tasks, including tasks in which semantics was task-irrelevant. Conclusions: Impaired prosody judgment in hearing-impaired individuals likely stems from auditory processing deficits, leading to poorer performance than their normal-hearing peers. The two groups showed different exploratory patterns of cross-channel interference: in the CI group, the face task showed no statistically detectable interference, whereas in the HA group, semantic inconsistency was associated with lower performance even when semantics was task-irrelevant. These patterns are potentially consistent with device-related differences in access to auditory and visual emotion cues, and we interpret them within a cross-channel plasticity framework as hypotheses for future study. These findings support individualized speech therapy tailored to specific perceptual strategies, enhancing targeted rehabilitation.