A Secure Federated Learning Algorithm for Emotion Recognition Towards Multimodal Speaker Signals on the Client Side

With the widespread application of multimodal data in dialog emotion recognition, effectively integrating text, audio, and visual information while addressing data heterogeneity across multiple clients and ensuring user privacy has become a key research challenge. This paper integrates a Transformer self-distillation model with attention scores and federated learning algorithms to propose a multimodal emotion recognition framework. The framework employs intra-modal and inter-modal Transformers to capture multimodal interactions, enhances modality representations through attention weights, and incorporates a federated learning structure to safeguard data privacy. A global model distance-weighted aggregation strategy is introduced to mitigate model bias caused by heterogeneous data. Experimental results on the IEMOCAP dataset demonstrate that the proposed framework achieves superior emotion recognition accuracy and exhibits more stable model convergence compared to existing baseline models.