Abstract To address the issues of low detection accuracy for small-sized defects and high model complexity in steel surface defect detection tasks, in this paper, we propose a lightweight model called EL-DETR for steel surface defect detection. Firstly, a dynamic sparse attention module is proposed to adaptively select key feature regions and achieve precise localization in the spatial dimension, effectively suppressing background noise and redundant information interference. Secondly, the multi-scale fusion attention MFA module is employed, which effectively integrates feature information from different levels while significantly reducing the model’s parameter count and computational load, thereby enhancing the detection capability of multi-scale targets. Finally, the content refined attention module is proposed to further enhance the model’s focus and representation capability for small-sized defect regions. To evaluate the effectiveness of the proposed model, we conducted experiments on the NEU-DET and GC10-DET steel surface defect datasets. The experimental results indicate that on the GC10-DET dataset, the parameter count of EL-DETR is reduced by 6.2% compared to the baseline model RT-DETR-L, the floating-point operations are reduced by 31.1%, the mean average precision is improved by 4.4%, and the FPS is increased by 25%.