Detecting small objects in uncrewed aerial vehicle (UAV) remote sensing imagery remains challenging, as targets often occupy only a few pixels and are easily overwhelmed by cluttered backgrounds, occlusions, and repetitive textures, leading to missed detections and background-induced false alarms. To address these challenges, we propose scene-prior guided cross-partial mixing network (SPM-DETR), an end-to-end framework tailored for small object detection in UAV remote sensing imagery. We first develop the frequency-augmented local context network, a backbone that enhances high-frequency feature representations and facilitates efficient long-range feature interaction through frequency-domain modulation. Building on this, we design scene-prior guided sparse intra-scale feature interaction, a module that suppresses background-dominated responses in both channel and spatial dimensions. Finally, the cross-partial mixing network performs efficient cross-partial mixing and semantic-detail alignment, specifically optimizing the preservation of shallow, high-resolution spatial cues during multilevel feature fusion. Experiments on the VisDrone-DET show that SPM-DETR achieves 54.1% mAP $_{50}$ and 34.4% mAP $_{50:95}$ on the validation set (+6.2 and +5.0 points over the RT-DETR-R18 baseline), and 43.0% mAP $_{50}$ and 25.7% mAP $_{50:95}$ on the test-dev set. SPM-DETR further outperforms other state-of-the-art detectors while using only 59.5% of the parameters of the baseline. Cross-dataset evaluations on NWPU VHR-10, AI-TOD, and SIMD further demonstrate robust generalization across varying object scales and complex background distributions in UAV-based Earth observation scenarios.