Inverse Feature Consistency Federated Unlearning for Vision-Language Model

Vision-Language Models (VLMs), with their advantages in vision and language processing, exhibit immense potential in mobile intelligent systems. Integrating federated learning with parameter-efficient fine-tuning of VLMs helps address data heterogeneity challenges. However, existing methods mainly focus on task-specific patterns, neglecting the impact of general features, such as background information and low-quality data, which weakens the model's ability to generalize when handling data from different sources and dealing with fluctuations in quality. To tackle these challenges, we propose Inverse Feature Consistency Federated Unlearning (IFCFU) for VLM, comprising three components: 1) Feature Consistency Federated Learning (FCFL) aligns fine-tuned features with pre-trained features through constraints to ensure the preservation of general features; 2) Pseudo-label Low-quality Data Detection (PLDD) identifies potential low-quality data through model quality assessment and pseudo-label generation; 3) Inverse Feature Consistency Unlearning (IFCU) distances low-quality data features from optimal model features to eliminate the negative impact and restores training with pseudo-labels. Evaluations on StanfordCars show that FCFL increased accuracy by 4.97% and 29.88% under normal data and low-quality data configurations, respectively. PLDD identified over 90.00% of low-quality data, while IFCU improved the global model's accuracy by 4.43% with 80% low-quality data.