Knowledge Guided Articulatory and Spectrum Information Fusion for Obstructive Sleep Apnea Severity Estimation

Numerous studies have demonstrated that speech analysis during wakefulness is a non-invasive and convenient method for Obstructive sleep apnea (OSA) screening. However, the inherent differences in upper airway structure and function between wakefulness and sleep limit the effectiveness of OSA assessments based on vowels and phonemes employed in existing studies. To address this challenge, we propose the design of controlled articulations that more accurately simulate upper airway obstruction during sleep, offering a more comprehensive reflection of the pathological changes in upper airway anatomy and function in individuals with suspected OSA. Specifically, we constructed a Mandarin Chinese controlled articulation dataset, consisting of speech recordings from 301 male adult participants who underwent polysomnography (PSG) monitoring at a sleep center. Drawing on domain knowledge, we thoroughly investigated articulations associated with upper airway collapse, including vowels, pharyngeals, and nasals, and identified interpretable optimal articulations using SHapley Additive Explanations (SHAP). Furthermore, we introduced a dual-stream fusion model, PTF-Net, which employs the Paralinguistic Acoustic Feature stream (PAF-Stream) to extract the physical attributes of speech and the Transfer Learning-based Spectrogram Feature stream (TLE-Stream) to capture the nonlinear features of upper airway dynamics. The Swin Transformer is utilized to integrate both local and global information from various articulations. Experimental results demonstrate that the knowledge-guided PTF-Net model outperforms existing methods in the assessment of OSA severity. The knowledge-guided PTF-Net model outperforms existing methods by 5.1% in Area Under the Curve (AUC) and 5.8% in Unweighted Average Recall (UAR) for OSA severity assessment. In addition, we revealed that the proposed deep embedding of controlled articulation could differentiate between different types of obstruction sites identified by drug-induced sleep endoscopy (DISE), suggesting its potential as a novel digital biomarker for upper airway assessment in OSA patients. This study enhances the understanding of speech-based OSA screening and paves the way for its broad clinical application.