人工智能
计算机科学
融合
领域(数学)
传感器融合
互补性(分子生物学)
模式
多模态
机器学习
代表(政治)
过程(计算)
模态(人机交互)
图像融合
多模式学习
融合机制
特征(语言学)
深度学习
编码器
模式识别(心理学)
特征学习
自编码
可视化
模式治疗法
多通道交互
融合规则
作者
ZHANG Hucheng,LI Leixiao,LIU Dongjiang
标识
DOI:10.3778/j.issn.1673-9418.2403083
摘要
Although the powerful learning ability of deep learning has achieved excellent results in the field of single-modal applications, it has been found that the feature representation of a single modality is difficult to fully contain the complete information of a phenomenon. In order to break through the obstacles of feature representation on a single modality and make greater use of the value contained in multiple modalities, scholars have begun to propose the use of multimodal fusion to improve model learning performance. Multimodal fusion technology is to make the machine use the correlation and complementarity between modalities to fuse into a better feature representation in text, speech, image and video, which provides a basis for model training. At present, the research of multimodal fusion is still in the early stage of development. This paper starts from the hot research field of multimodal fusion in recent years, and expounds the multimodal fusion method and the multimodal alignment technology in the fusion process. Firstly, the application, advantages and disadvantages of joint fusion method, cooperative fusion method, encoder fusion method and split fusion method in multimodal fusion are analyzed. The problem of multimodal alignment in the fusion process is expounded, including explicit alignment and implicit alignment, as well as the application, advantages and disadvantages. Secondly, it expounds the application of popular datasets in multimodal fusion in different fields in recent years. Finally, the challenges and research prospects of multimodal fusion are expounded to further promote the development and application of multimodal fusion.
科研通智能强力驱动
Strongly Powered by AbleSci AI