A Self Supervised Transformer Based Framework for Brain Tumor Detection and Segmentation Using MRI

Brain tumor diagnosis through MRI is a critical clinical task where annotation scarcity and limited computational resources constrain the adoption of deep learning. Existing convolutional neural network approaches rely heavily on large labeled datasets and lack robustness across imaging centers. This paper proposes a self-supervised transformer-based framework built on DINO (self-distillation with no labels) for brain tumor detection and segmentation. A Vision Transformer (ViT-Small) serves as the teacher network and a lightweight CNN-Transformer hybrid acts as the student, enabling knowledge distillation without pixel-level annotations. Domain adaptive pre-training is combined with selective layer adaptation, which freezes early feature extraction layers and fine-tunes only semantic mid- to high-level layers, reducing computational cost by approximately 40% compared to full fine-tuning. A weakly supervised segmentation inference pipeline uses CNN activation maps and transformer attention heatmaps for tumor localization. While the segmentation network is trained with ground-truth Dice loss, inference employs weak visual cues. Evaluated on the Kaggle Brain MRI dataset, the proposed framework achieves a Dice score of 67.4% for segmentation, outperforming a supervised CNN baseline by 5.3 percentage points, with an AUCROC of 0.75 and F1-score of 0.83 for classification. The entire pipeline runs on CPU-only hardware under 5.5GB RAM, demonstrating clinical feasibility in resource-constrained settings. These results establish self-supervised vision transformers as a viable and computationally efficient approach for brain tumor analysis with limited labeled data.