Existing self-supervised deep learning methods for single-pixel imaging (SPI) often struggle to simultaneously achieve high reconstruction quality and computational efficiency. To address this limitation, this paper proposes FDE-Net, an efficient self-supervised SPI algorithm based on frequency-domain enhancement. FDE-Net adopts a single-path encoder–decoder architecture and reduces computational complexity by implementing feature upsampling through a lightweight efficient up-convolution module. To ensure high-fidelity target reconstruction, a frequency-domain brightness adaptive correction module is introduced alongside a multi-scale attention fusion strategy to enhance the network’s feature representation capability. Experimental results on the Set11 dataset demonstrate that FDE-Net achieves the best reconstruction quality among all compared methods. At a 25% sampling rate, it yields an average PSNR improvement of 2.1 dB over the algorithm SDD-Net while reducing reconstruction time by approximately 33.3%, effectively balancing reconstruction quality and computational efficiency.