EmoSENSE: Modeling Sentiment-Semantic Knowledge with Hierarchical Reinforcement Learning for Emotional Image Generation

Emotional image generation aims to create images that effectively reflect target emotions. A fundamental challenge in this task is the affective gap, which refers to the discrepancy between visual content and emotional states perceived by users. Existing methods generally assume strong and explicit associations between target emotions and specific objects (e.g., “monster” and “fear”), which limits their generalization ability when encountering uncommon emotion-object pairs. This limitation stems from two main factors: 1) Most existing approaches primarily focus on semantic alignment without explicitly modeling how emotions influence visual attributes such as brightness and colorfulness; 2) diffusion-based image generation methods have limited capability in handling diverse sentiment-semantic pairs. To address these challenges, we propose EmoSENSE, a novel hierarchical fuzzy reinforcement learning framework for the emotional image generation task. EmoSENSE consists of a high-level module and a low-level module, working collaboratively in a hierarchical structure to inject sentiment-semantic knowledge into emotional images. The high-level module quantifies sentiment-semantic correlations within a unified emotional space, connecting emotions to visual attributes. The low-level module refines this connection by optimizing a fuzzy-logic-based mapping between emotions and visual attributes through reinforcement learning, enabling flexible adaptation to diverse emotion-object pairs. Extensive qualitative and quantitative experiments on public dataset demonstrate that EmoSENSE significantly enhances both the visual quality and emotional expression ability of the generated images, achieving a 12.21% higher EmoAccuracy-8 classes than the previous state-of-the-art methods. https://github.com/forever3600/EmoSENSE.