Dual-feature adaptive framework for multimodal disinformation detection

Abstract The spread of disinformation on online social media has caused massive concern. Existing disinformation detection methods neglect the diverse compositional forms of tweets in real-life scenarios, making them less applicable and effective in social media settings. Meanwhile, these methods use pattern cues but overlook important aspects such as syntax, lexicon, and shallow visual semantics, and lack attention to factual content such as time, place, and person relay in both text and images, thus failing to fully explore features of disinformation and limiting detection accuracy. Furthermore, with the popularity of large language models (LLMs), the tweets generated by these models make the style of disinformation more subtle. Since existing datasets are mostly human-generated and lack style diversity, it results in weak detection capabilities of methods trained on these datasets. To address these challenges, a dual-feature adaptive framework for multimodal disinformation detection is proposed. The framework first using a similarity-based algorithm adaptively handles different tweet forms. It then enhances pattern features by bridging multimodal output from single-modal pretrained modal, and factual features are subsequently extracted using a zero-shot method based on a large vision language model. Finally, an expert network aggregates and reweights the dual-feature representation for tweets using an LLM-text detector in gating strategy. This paper also presents two multimodal disinformation datasets that include both LLM-generated and human-generated tweets reflecting real-world scenarios. The true tweets in datasets are diverse in style, while the fake tweets are more misleading. Experimentally verified, the proposed method outperforms baseline methods by an accuracy of 1.04% and 0.72% on typical datasets while also achieving a minimum accuracy drop of 0.65% and 0.87% on the proposed dataset.