GenHMD: Enhancing Hateful Meme Detection with Generated Rationale from Multimodal Large Language Models

Detecting harmful memes plays a crucial role in safeguarding social media users and maintaining a healthy network ecosystem. However, the ability of multimodal large language models(MLLMs) in harmful meme detection tasks remains to be fully developed. Fine-tuning MLLMs is prohibitively expensive due to the large number of parameters, making it essential to leverage these powerful models more efficiently rather than simply fine-tuning them. In this work, we propose GENHMD, a generated rationale-enhanced detection framework, which generates rationales by prompting MLLMs. Our key insight is that MLLMs, pre-trained on massive amounts of data containing rich knowledge, can enhance harmful meme detection by eliciting specialized knowledge. Specifically, inspired by the powerful capacity of MLLMs for text generation and reasoning, we first employ MLLMs to generate credible reasons through chain-of-thought reasoning. To further promote collaboration between harmfulness rationales and the multimodal information inherent in memes, we leverage a smaller multimodal model to learn the attention weights between different modal features using the attention mechanism. Comprehensive experiments on two public meme datasets demonstrate that GENHMD achieves outstanding performance in harmful meme detection. Further analysis shows that the generated rationales substantiate the explainability of the detection results.