Robust multimodal emotion recognition under missing and incomplete data with cross-modal regeneration

作者
Behzad Mahaseni,Naimul Khan
标识
DOI:10.70401/ec.2026.0023
摘要

Aims: Multimodal emotion recognition (MER) can outperform unimodal approaches by integrating complementary information from multiple sources. However, real-world applications often involve incomplete or missing modalities, reducing the reliability of existing MER models. This study proposes a framework that remains robust under missing-modality conditions while preserving the advantages of multimodal integration. Methods: To address this challenge, we propose a cross-modal latent regeneration and attention long short-term memory (CMLR-ALSTM) framework. The framework combines pretrained variational autoencoder encoders with residual projection networks, optimized with L2 loss, to achieve stable latent-space alignment across modalities. Regenerated and available latent representations are integrated using a cross-modal attention mechanism and processed by an LSTM to model temporal dependencies and improve multimodal fusion under incomplete data. Results: The proposed framework was evaluated on three benchmark datasets under complete, partially missing, and completely missing modality scenarios. Experimental results demonstrate that CMLR-ALSTM achieves up to 17.22% improvement under missing-modality conditions while maintaining competitive performance under complete-modality settings. The results also show that the proposed latent regeneration strategy effectively preserves cross-modal relationships and robust latent representations. Conclusion: The experimental results confirm the effectiveness of the proposed framework, particularly in realistic environments where data availability is inconsistent. By leveraging CMLR to reconstruct missing representations and modelling temporal dependencies through LSTM, the proposed approach provides a more robust and reliable MER framework for practical deployment. The results across different modality combinations demonstrate its ability to generalize across heterogeneous multimodal settings.

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
longyuyan完成签到,获得积分0
刚刚
12完成签到,获得积分20
3秒前
卓卓发布了新的文献求助10
3秒前
3秒前
ZTK完成签到,获得积分10
3秒前
咻咻发布了新的文献求助10
4秒前
Jasper的应助被朴实的以南采纳,获得10
4秒前
yyyyy发布了新的文献求助10
5秒前
帆帆完成签到,获得积分10
5秒前
务实凌旋完成签到,获得积分10
6秒前
小刘爱科研完成签到,获得积分10
6秒前
科目三的应助被LY采纳,获得10
8秒前
8秒前
8秒前
欣喜成仁完成签到,获得积分10
8秒前
星星完成签到,获得积分10
8秒前
chen完成签到 ,获得积分10
9秒前
9秒前
Mr.Bad完成签到,获得积分10
9秒前
tuihuo完成签到,获得积分10
9秒前
10秒前
合适忆枫发布了新的文献求助10
12秒前
自然白安发布了新的文献求助10
12秒前
大佬发布了新的文献求助10
12秒前
哭哭完成签到 ,获得积分10
13秒前
nan发布了新的文献求助10
15秒前
优秀含之完成签到,获得积分20
15秒前
比奇堡完成签到,获得积分10
16秒前
17秒前
111发布了新的文献求助10
17秒前
佩楠发布了新的文献求助10
17秒前
19秒前
19秒前
19秒前
20秒前
21秒前
春衫发布了新的文献求助10
23秒前
^O^发布了新的文献求助30
23秒前
24秒前
25秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 2000
Handbook of Sport Coaching Research 1000
Orthognathic Surgery - 2 Volume Set 500
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
基于结构调控的功能化水诱导发电材料与器件研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8193514
求助须知:如何正确求助?哪些是违规求助? 9579686
关于积分的说明 21866883
捐赠科研通 7793152
什么是DOI,文献DOI怎么找? 3437297
关于科研通互助平台的介绍 2560725
邀请新用户注册赠送积分活动 2484176