计算机科学
模式
代表(政治)
人工智能
成对比较
特征学习
奇异值
矩阵表示法
正规化(语言学)
机器学习
外部数据表示
奇异值分解
功能(生物学)
源代码
模式识别(心理学)
依赖关系(UML)
理论(学习稳定性)
多模态
缩小
矩阵范数
编码(集合论)
多模式学习
理论计算机科学
矩阵分解
作者
Xiaohao Liu,Xiaobo Xia,See Kiong Ng,Tat‐Seng Chua
标识
DOI:10.1109/tpami.2026.3675685
摘要
Multimodal representation learning seeks to create a unified representation space by integrating diverse data modalities to improve multimodal understanding. Traditional methods often depend on pairwise contrastive learning, which relies on a predefined anchor modality, restricting alignment across all modalities. Recent advances have investigated the simultaneous alignment of multiple modalities, yet several challenges remain, such as limitations imposed by fixed anchor points and instability arising from optimizing the product of singular values. To address the challenges, in this paper, we propose Principled Multimodal Representation Learning (PMRL), a novel framework that achieves simultaneous alignment of multiple modalities without anchor dependency in a more stable manner. Specifically, grounded in the theoretical insight that full alignment corresponds to a rank-1 Gram matrix, PMRL optimizes the dominant singular value of the representation matrix to align modalities along a shared leading direction. We propose a softmax-based loss function that treats singular values as logits to prioritize the largest singular value. Besides, instance-wise contrastive regularization on the leading eigenvectors maintains inter-instance separability and prevents representation collapse. Extensive experiments across diverse tasks demonstrate PMRL's superiority compared to baseline methods.
科研通智能强力驱动
Strongly Powered by AbleSci AI