计算机科学
语音识别
语音处理
人工智能
人工神经网络
说话人日记
说话人识别
信号处理
模式识别(心理学)
语音活动检测
自适应滤波器
说话人识别
噪音(视频)
作者
Hang Chen,Maokui He,Jun Du,Sabato Marco Siniscalchi,Li-Juan Liu,Genshun Wan,Chin‐Hui Lee
出处
期刊:
[Institute of Electrical and Electronics Engineers]
日期:2026-01-01
卷期号:34: 3146-3158
标识
DOI:10.1109/taslpro.2026.3698952
摘要
Traditional audio-visual speaker diarization (AVSD) approaches exhibit limited robustness to cross-modal heterogeneity and complex inter-speaker interactions, particularly under dynamic, unconstrained real-world conditions. To overcome these limitations, we propose a novel neural architecture, termed the hierarchical modality–speaker adaptive network (HMSA-Net), which integrates two sequentially structured modules and an ad-hoc multi-stage training strategy. The first consistency-gated inter-modal attention (CGIMA) module dynamically estimates cross-modal synchrony between audio and visual embeddings and adaptively regulates their mutual influence during feature fusion, thereby mitigating modality mismatch. The second dense inter-speaker attention (DISA) module explicitly captures complex inter-speaker relationships by applying multi-head attention from a target speaker representation to a densely aggregated bank of non-target speaker embeddings, enabling fine-grained discrimination in overlapping speech conditions without enforcing a fixed upper bound on the number of speakers. To further enhance optimization stability, a multi-stage optimization (MSO) scheme is introduced, which consistently achieves lower convergence loss than end-to-end training. Extensive evaluations on standard AVSD benchmarks demonstrate that CGIMA effectively suppresses modality-specific noise while amplifying complementary cross-modal cues, resulting in more robust fused representations. Meanwhile, DISA improves frame-level speaker discrimination by modeling dense cross-speaker dependencies. As a result, HMSA-Net trained with MSO achieves state-of-the-art performance on the AMI, MISP2022, and AVA-AVD.
科研通智能强力驱动
Strongly Powered by AbleSci AI