An Investigation of Listening Effort with Concurrent fNIRS and Pupillometry

The pupillary response can be used to index listening effort during speech recognition tasks, such that pupil size increases with increasing cognitive demand. We have previously shown that middle frontal (MFG) and middle temporal gyri (MTG) are engaged when listening conditions are difficult (Defenderfer et al., 2021, NeuroImage) and recognition of simulated cochlear implant (CI) speech (i.e., vocoded) require more effort (as indexed by the pupillary response) than speech in quiet (Defenderfer et al., 2020, VSS). In the current study, we examined the influence of semantic context on effortful listening. We collected concurrent neural (near-infrared spectroscopy) and pupil data from 41 participants. We manipulated the difficulty of sentences in two ways: by presenting sentences with or without semantic context (High-, Low-Predictability) and by spectrally degrading the speech quality (Non-Vocoded, Vocoded). We predicted a decrease in effort when context can be used to predict meaning in challenging (i.e., vocoded) condition and that the degree of effort would be related to activation in the MFG and MTG. Replicating our previous study, vocoded speech resulted in larger pupil sizes than non-vocoded. Neural data showed (1) increased activation over the MFG for vocoded/high-predictable speech, suggesting that top-down mechanisms are engaged when context can be used to mitigate listening effort, and (2) an elevated response in the MTG during high-predictable speech conditions, indicating its role in accessing and retrieving semantic representations. Lastly, positive correlations between pupil and neural data were observed during high-predictable/vocoded conditions in the MFG and MTG, providing evidence that these two regions are directly involved in effortful speech processing. Thus, by integrating physiological (pupillometry) and neural measures, we present novel data on the dynamics of effort and how they are related to neural processes.