Automated karyotyping and structural anomaly detection through a hybrid multi-stage deep learning framework integrating chromosome detection, pairwise classification, and autoencoder-based analysis

Manual karyotype analysis remains a labor-intensive and expertise-dependent process that requires visual interpretation of chromosome banding patterns under the microscope. Although recent studies have applied machine learning to isolated stages of chromosome detection, segmentation, or classification, few approaches address the complete cytogenetic workflow in a clinically aligned manner. This work proposes a stability-aware, multi-stage deep learning framework for semi-automated karyotype generation and chromosomal anomaly screening. The pipeline integrates: (i) an overlapped-aware quality-control gate to exclude morphologically unreliable metaphases; (ii) YOLOv8-based single-chromosome detection; (iii) homologous-pair assignment using both a unified YOLOv8 route and a hybrid YOLOv8 + fine-tuned ResNet-50 configuration across 24 chromosome classes; (iv) automatic karyotype assembly; and (v) unsupervised structural anomaly screening via a convolutional autoencoder trained exclusively on normal chromosomes. Experimental validation demonstrated 98.03% detection accuracy (mAP@0.5 = 0.985) and 94.01% validation accuracy for homologous-pair classification. Under sequential modeling, the propagated end-to-end numerical accuracy reached 92.21% for the hybrid configuration and 94.01% for the unified YOLOv8 configuration. The autoencoder exhibited clear separation between normal and structurally anomalous reconstruction-error distributions, enabling effective unsupervised screening without requiring extensive labeled pathological datasets. By explicitly modeling cumulative inter-stage dependency and enforcing morphological integrity through an upstream overlapped filter, the proposed framework provides a clinically interpretable and operationally coherent assistant system. The results demonstrate that stability-aware pipeline design can support both numerical and structural anomaly detection while reducing manual workload and preserving diagnostic transparency.