计算机科学
图像融合
人工智能
保险丝(电气)
光学(聚焦)
特征(语言学)
融合
图像(数学)
语义学(计算机科学)
代表(政治)
计算机视觉
模式识别(心理学)
哲学
工程类
物理
光学
电气工程
政治
程序设计语言
法学
语言学
政治学
作者
Zhijia Yang,Yunping Li,Xin Tang,Minghong Xie
标识
DOI:10.3389/fnbot.2024.1521603
摘要
Existing image fusion methods primarily focus on complex network structure designs while neglecting the limitations of simple fusion strategies in complex scenarios. To address this issue, this study proposes a new method for infrared and visible image fusion based on a multimodal large language model. The method proposed in this paper fully considers the high demand for semantic information in enhancing image quality as well as the fusion strategies in complex scenes. We supplement the features in the fusion network with information from the multimodal large language model and construct a new fusion strategy. To achieve this goal, we design CLIP-driven Information Injection (CII) approach and CLIP-guided Feature Fusion (CFF) strategy. CII utilizes CLIP to extract robust image features rich in semantic information, which serve to supplement the information of infrared and visible features, thereby enhancing their representation capabilities for the scene. CFF further utilizes the robust image features extracted by CLIP to select and fuse the infrared and visible features after the injection of semantic information, addressing the challenges of image fusion in complex scenes. Compared to existing methods, the main advantage of the proposed method lies in leveraging the powerful semantic understanding capabilities of the multimodal large language model to supplement information for infrared and visible features, thus avoiding the need for complex network structure designs. Experimental results on multiple public datasets validate the effectiveness and superiority of the proposed method.
科研通智能强力驱动
Strongly Powered by AbleSci AI