Road damage detection is crucial for maintenance and management. Timely and accurate detection improves traffic safety and extends the road service life. However, road damage in complex backgrounds is often characterized by large aspect ratios, multiple scales, and abrupt changes in direction, which greatly reduce detection accuracy. To solve the problem, this paper proposes MMR-DETR, a road damage detection network with cross fusion of multi-scale information. Specifically, for the multi-scale and large aspect ratio of damage in complex backgrounds, the encoder introduces multi-scale multi-head self-attention (M2SA) and multi-scale cross fusion (MCF) modules to learn damage information, enhancing feature representation and detection performance. Additionally, a redundant bounding box merging (RBBM) method is applied to improve localization accuracy by optimizing detection boxes. To evaluate the effectiveness of the proposed model, we conducted experiments on the UAPD, UAVPDD-2023, and RDD2022 datasets. The experimental results show that our model outperforms existing models in terms of accuracy, recall, and mAP@0.5, exhibiting excellent detection performance and generalization. The code is available at https://github.com/Lrc-1109/MMR-DETR.