YOLO-TeethSeg: a resource-efficient approach to multi-class teeth instance segmentation using lightweight YOLO-based models

Panoramic dental X-ray interpretation for full 32-tooth analysis remains time-consuming and expertise-dependent, highlighting the need for automated, resource-efficient instance segmentation tools deployable in low-resource clinical settings. This study presents, a lightweight deep learning framework utilizing three nano/compact YOLO segmentation models (YOLOv8n-seg, YOLOv9c-seg, YOLOv11n-seg) for automated detection and instance segmentation of all 32 teeth from panoramic radiographs. Transfer learning was applied on a publicly available Roboflow dataset comprising 594 annotated panoramic images (15,300 tooth instances; split: 353 train, 190 validation, 51 test). Models were trained on Google Colab’s free tier environment equipped with an T4 GPU (15 GB VRAM), Intel Xeon CPU, and 13 GB RAM. Performance was evaluated using box and mask precision, recall, and mean Average Precision (mAP) at IoU thresholds of 0.5 and 0.5–0.95. YOLOv9c-seg achieved the highest results (box mAP@50 = 0.960, mask mAP@50 = 0.961), while YOLOv11n-seg and YOLOv8n-seg offered superior inference speed (∼3.9–4.2 ms/image) with near-comparable accuracy, demonstrating an effective accuracy-efficiency trade-off. Model results show that lightweight single pass YOLO-based instance segmentation models achieve near state-of-the-art performance for multi-tooth segmentation while being computationally cheaper, enabling real-time clinical screening for diagnostic purposes. This study presents comparative analyzes of smallest YOLO segmentation models on dental X-ray images for discovering potential low resource applications on chairside and mobile diagnostic devices.