A Hybrid Clinical Knowledge-Driven Transformer for Breast Ultrasound Video Classification

Early diagnosis of breast cancer is critical for reducing mortality rates. Dynamic ultrasound videos contain rich tumor-specific features, offering valuable information for clinical diagnosis. In standard clinical practice, sonographers typically first identify keyframes before scanning the surrounding area of it to gather more information. Previous research based on ultrasound videos has been devoted to temporal modeling while neglecting the contribution of keyframes to tumor diagnosis. In this paper, we propose a two-stage hybrid network, Hybrid Keyframe-Guided Video Transformer (HKVT), to model both static keyframe and dynamic video information in breast ultrasound videos. In the first stage, the model uses a multi-instance learning paradigm to construct an efficient video classification model that automatically identifies keyframes using self-attention scores. In the second stage, the embedding tokens of the keyframe are extracted, and a keyframe-guided transfromer block is constructed for ultrasound video classification. Specifically, we designed a Keyframe-Guided Temporal Attention Module and a Keyframe-Guided Spatial Co-Attention Module to incorporate static keyframe features alongside dynamic video features. We evaluated the proposed model on an internal dataset of 342 patients and an external test dataset of 119 patients. The HKVT model achieved an AUC of 0.921 on the internal dataset and 0.901 on the external test dataset, outperforming other state-of-the-art models. Furthermore, our model demonstrated robust performance on 242 multi-center test cases, outperforming other models by at least 2.1% in AUC. These results demonstrate the superiority of our approach for breast ultrasound video classification.