Enhancing Speaker Verification with Layer-Wise Mixture-of-Experts on Pre-Trained Models

In this paper, we propose a novel speaker verification (SV) method that integrates speech pre-trained models (PTMs) with a Mixture-of-Experts (MoE) mechanism. The method first extracts multilayer representations from the PTMs and inserts an MoE module after each Transformer layer. Through dynamic routing, the MoE module performs input-adaptive feature transformations, which in turn enables implicit feature selection under varying domain conditions. We then design a cascaded residual gating mechanism to propagate key shallow-layer acoustic cues and prevent their attenuation in deeper representations. Furthermore, we introduce a cross-attention fusion module that dynamically integrates multilayer features along the layer dimension, thereby jointly maintaining the robustness of shallow features and the discriminability of deep features. The fused representations are finally fed into a downstream SV network to generate highly discriminative speaker embedding. Experiments on the VoxCeleb1-O test set show that, with the Base and Large versions of PTMs, our method achieves equal error rates of 0.628% and 0.487%, respectively, which demonstrates the effectiveness and superiority of the proposed method.