Abstract Balancing detection accuracy and computational efficiency is paramount for effective pavement distress detection using unmanned aerial vehicles (UAVs), especially given the resource constraints of edge devices. However, existing UAV-based visual methods often falter in handling complex backgrounds, diverse target sizes, and struggle to extract critical features efficiently, resulting in high computational costs. To tackle these issues, this paper proposes a lightweight detection algorithm, LS-YOLOv11. The algorithm proposes the CGLU Former Block, enabling the model to extract fine-grained features from road surfaces with enhanced precision. The introduction of large separable kernel attention to modify the spatial pyramid pooling layer allows the model to capture multi-scale context and long-range dependencies effectively, improving its adaptability to various distress sizes and shapes. Moreover, a lightweight feature fusion structure is devised with a dual-branch pooling approach, reducing computational overhead while maintaining key feature integrity. Additionally, a lightweight shared detection head is proposed to eliminate parameter redundancy and enhance cross-scale semantic consistency. Experimental results on the UAV-PDD2023 dataset show that LS-YOLOv11 achieves a mAP@50 of 87.1%, a 5.1% improvement over YOLOv11n. The algorithm also reduces parameters by 23.3% and GFLOPs by 17.5%, achieving high accuracy with minimal complexity. On the UAPD dataset, LS-YOLOv11 achieves a 2.6% improvement in mAP@50, demonstrating its superior generalization capability. Furthermore, it attains an inference speed of 95 fps on edge devices, highlighting its efficiency and suitability for real-world UAV-based pavement distress detection applications.