Robust frame-level speaker localization guided by multi-channel speech enhancement and inter-channel phase-difference losses

In the presence of room reverberation and background noise, the performance of frame-level speaker localization is severely limited. To address this challenge, this study performs multi-channel speech enhancement based on complex spectral mapping (CSM), followed by direction-of-arrival (DOA) estimation using weighted generalized cross-correlation with phase transform (GCC-PHAT). The proposed approach differs from prevailing deep learning methods that operate on multi-channel inputs directly for speaker localization. This study initially investigates multi-input single-output (MISO) based speech enhancement models with two loss functions and then extends to multi-input multi-output (MIMO) based modeling for conceptual and computational efficiency. The results demonstrate that the phase estimates obtained from CSM models are reliable for frame-level DOA estimation and MIMO systems outperform MISO systems. In addition, the study proposes new multi-channel loss functions for MIMO systems that incorporate phase differences in order to better preserve inter-channel phase relations, which is key to accurate sound localization. Systematic evaluations with multiple microphone array geometries using both simulated and recorded room impulse responses, as well as real recordings, demonstrate that the proposed model yields excellent frame-level speaker localization results in reverberant and noisy environments and outperforms related methods by a large margin, even surpassing their utterance-level results.