DBHF-RT-DETR: An Enhanced RT-DETR with Dual-Boundary Regression and Hybrid Frequency–Spatial Attention for Gallstone Detection in Ultrasound Images

Accurate detection of gallstones in ultrasound images is critical for clinical decision-making but remains challenging due to blurred boundaries, the difficulty of recognizing small objects, and severe speckle noise. To overcome these issues, we propose an enhanced end-to-end medical image detection framework, Dual-Boundary and Hybrid-Formers RT-DETR (DBHF-RT-DETR). First, to alleviate inaccurate localization in the presence of indistinct boundaries and low-overlap regions, we introduce a dual-boundary constrained regression loss, Dual-Boundary Intersection over Union (DBIoU), which strengthens the model’s boundary discrimination capability and robustness. Second, to address the challenge of small objects recognition in ultrasound images, we design Convolution-to-Hybrid-Formers (Conv2HybridFormers), which enhances multi-scale semantic and fine-grained feature representations by integrating the multi-scale convolutional aggregation of Local Feature Aggregation Module (LFAM) with the convolutional self-attention fusion of Hybrid-Formers Module (HFM). Furthermore, to suppress noise interference and amplify lesion-related responses, we propose the Frequency–Spatial Attention Residual (FSAR) module, which achieves joint attention enhancement in both the frequency and spatial domains. Systematic experiments and ablation studies were conducted on a self-built gallstone ultrasound image dataset and two public datasets. The experimental results show that, compared to the baseline model Real-Time Detector of Transformer (RT-DETR), DBHF-RT-DETR achieves a 2.2 percentage point increase in mean Average Precision at Intersection over Union threshold of 0.5 (mAP@0.5) on our in-house dataset, reaching 74.7%; on the public Gallstones-SDGIT dataset, mAP@0.5 improves by 2.8 percentage points to 73.7%; on the UIdataGB-Gallstones dataset, the mAP@0.5 increased from 66.2% to 67.3%, representing a 1.1 percentage point improvement. These results demonstrate that the proposed method achieves consistent performance improvements across different data distributions and levels of complexity, highlighting its effectiveness and robustness in the medical ultrasound gallstone detection task.