计算机科学
人工智能
特征(语言学)
钥匙(锁)
图像融合
突出
代表(政治)
噪音(视频)
图像(数学)
语义学(计算机科学)
特征提取
模式识别(心理学)
判决
特征学习
情报检索
计算机视觉
上下文图像分类
任务(项目管理)
融合
语义特征
语义鸿沟
语义计算
质量(理念)
深度学习
可视化
利用
传感器融合
机器学习
学习迁移
语义分析(机器学习)
自然语言处理
作者
Wenyu Shao,Hongbo Liu,Yunchuan Ma,Ruili Wang
标识
DOI:10.1109/tmm.2026.3668472
摘要
Existing text-driven infrared and visible image fusion approaches often rely on textual information at the sentence level, which can lead to semantic noise from redundant text and fail to fully exploit the deeper semantic value of textual information. To address these issues, we propose a novel fusion approach named Entity-Guided Multi-Task learning for infrared and visible image fusion (EGMT). Our approach includes three key innovative components: (i) A principled method is proposed to extract entity-level textual information from image captions generated by large vision-language models, eliminating semantic noise from raw text while preserving critical semantic information; (ii) A parallel multi-task learning architecture is constructed, which integrates image fusion with a multi-label classification task. By using entities as pseudo-labels, the multi label classification task provides semantic supervision, enabling the model to achieve a deeper understanding of image content and significantly improving the quality and semantic density of the fused image; (iii) An entity-guided cross-modal interactive module is also developed to facilitate the fine-grained interaction between visual and entity-level textual features, which enhances feature representation by capturing cross-modal dependencies at both inter-visual and visual-entity levels. To promote the wide application of the entity-guided image fusion framework, we release the entity-annotated version of four public datasets (i.e., TNO, RoadScene, M3FD, and MSRS). Extensive experiments demonstrate that EGMT achieves superior performance in preserving salient targets, texture details, and semantic consistency, compared to the state-of-the-art methods. The code and dataset will be publicly available at https://github.com/wyshao-01/EGMT.
科研通智能强力驱动
Strongly Powered by AbleSci AI