OmniFuse: Composite Degradation-Robust Image Fusion with Language-Driven Semantics

Existing image fusion methods struggle to accommodate composite degradation and do not support users flexibly modulating the semantic objects of interest. To address these challenges, this study proposes a composite degradation-robust image fusion framework with language-driven semantics, called OmniFuse. Firstly, OmniFuse establishes a novel multi-modal information fusion paradigm based on the latent diffusion model (LDM). By projecting the information fusion function into the latent space of the LDM, the information fusion process is seamlessly integrated with the diffusion process. Thus, OmniFuse fully leverages the powerful generative capabilities of LDM to eliminate composite degradation, thereby achieving highly robust image fusion. Secondly, OmniFuse develops a language-driven controllable fusion strategy to strengthen fusion flexibility. It employs a language-driven feature fusion module (LFFM) to receive the specified localization priori, dynamically aggregating multi-modal features. Within LFFM, a visual enhancement regularization is introduced to highlight objects of interest for capturing perceptual attention, while reverse semantic driving is established to strengthen their semantic attributes. Together, the visual and semantic constraints can implicitly correct the imperfect localization priori, further refining the accuracy of language-driven control. Extensive experiments demonstrate the omnipotent performance of OmniFuse, with significant advantages in robustness and flexibility compared to state-of-the-art methods. The code is publicly available at https://github.com/HaoZhang1018/OmniFuse.