The Role of Auditory and Visual Modality in Perception of Focus in Mandarin Chinese

Purpose: Speech perception is a complex process that involves multiple sensory modalities. Despite our intuitions of speech as something we hear, accumulating evidence has shown that speech perception is not solely dependent on the auditory modality. While it is well established that auditory and visual cues can both help listeners perceive focus, the latter is not established in Mandarin, and the relative contribution of these cues is not established at all. The current study investigated Mandarin listeners' integration of auditory and visual cues in the interpretation of focus in noise-degraded speech, through a question–answer appropriateness rating experiment. Method: To explore the effectiveness and relative contribution of auditory and visual modality in the interpretation of Mandarin focus, participants did a question–answer appropriateness rating task involving subject focus, object focus, and broad focus. All the question–answer pairs were constructed in three modalities: audio only, visual only, and audiovisual. They were instructed to rate the appropriateness of the question–answer pairs. A babble noise was superimposed on the audio track for the audio only and audiovisual conditions. Results and Conclusions: Although auditory cues via prosodic prominence were an effective cue to interpreting focus, visual cues were proven more effective, at least with degraded audio. Overall, this research contributes to our understanding of the interaction between linguistic cues and sensory information during language comprehension, widens the range of languages included in this body of research, and provides important implications for future studies on focus processing in various linguistic contexts and communication settings. This, in turn, will deepen our understanding of the multimodal nature of language comprehension.