Consensus-Driven Distillation for Trustworthy Explanations in Self-Interpretable GNNs

Self-interpretable graph neural networks (SI-GNNs) provide built-in explanations, yet these explanations are often inconsistent across independently trained models of the same architecture. Our prior work identified redundancy-the inclusion of uninformative features in explanations-as a primary cause of this inconsistency, and demonstrated that averaging explanations from multiple models (Explanation Ensemble, EE) effectively suppresses the high-variance scores of redundant features, thereby improving both explanation consistency and accuracy. Despite its effectiveness, however, EE requires running multiple models to generate consensus explanations, resulting in prohibitive inference cost and incompatibility with standard single-model metrics. In this work, we propose Consensus-driven explanation Distillation (CD), a simple, general, and theoretically grounded framework that leverages ensemble-derived consensus explanations as supervision signals, teaching a single SI-GNN to assign importance scores more responsibly. Experiments across diverse benchmark datasets and SI-GNN frameworks demonstrate that CD retains the benefits of EE while overcoming its limitations, and that combining CD with EE yields additional gains. Our code is publicly available at https://github.com/ICDM-UESTC/ConsensusDistillation.