CausalProto-CLIP:Fine-Grained and Interpretable Vision-Language Modeling via Causal Prototypes

Although Contrastive Language-Image Pre-training (CLIP) models demonstrate exceptional performance in downstream tasks, their decision-making processes exhibit significant limitations in fine-grained part-level interpretability, with existing methods predominantly focusing on global features and struggling to precisely localize critical discriminative regions. To address this challenge, this paper proposes CausalProto-CLIP(CP-CLIP), an innovative framework that enhances both the fine-grained classification capability and interpretability of CLIP models through causal prototype-guided mechanisms. The framework consists of two main modules: Causal Negative Prototypes (CNP) for prototype-guided fine-tuning and CausalProto Chain (CPC) for interpretability. It employs a causal prototype extraction strategy to construct robust part-level representations from training images, using adaptive thresholding and causal intervention testing to identify positive prototypes with significant impacts on class predictions. The CNP module innovatively generates negative prototypes through controlled perturbations and causal validation, integrating them into the contrastive learning process with additional alignment and negative contrastive losses to reinforce causal reasoning, thereby significantly boosting the model's discriminative power, robustness, and generalization. Complementing this, the CPC module provides flexible interpretability through region detection, prototype matching, enhanced causal analysis with do-calculus and backdoor adjustment, and a trustworthiness scoring system, achieving a breakthrough from coarse-grained global explanations to finegrained part-level decision tracing with causal graphs. Experiments demonstrate that CP-CLIP maintains excellent classi-fication performance while precisely localizing decision-critical components, revealing how CLIP models utilize local features and providing crucial transparency for safety-critical deployment scenarios.