Hyperspectral imaging is widely used in remote sensing, biomedicine, and industrial inspection. Single-pixel spectral imaging reduces the measurement load through compressive acquisition and extends spectral coverage to non-visible wavelengths where array detectors are costly. However, existing approaches rely on iterative reconstruction algorithms, which impose substantial computational costs and limit real-time applications. Here, we propose a dual-DMD Fourier single-pixel imaging (DM-FSPI) method based on joint spatial–spectral Fourier modulation. Spatial encoding uses Fourier basis patterns, while spectral encoding is achieved by frequency-division multiplexing, mapping different spectral channels to distinct temporal carrier frequencies. A temporal Fourier transform separates all spectral channels, and an inverse Fourier transform reconstructs the spatial image at each wavelength, obviating iterative computation. This design simplifies both encoding and reconstruction, and enables high-channel-count hyperspectral acquisition at low computational cost. Simulations on the CAVE dataset confirm high spatial-spectral fidelity and strong noise robustness, while experiments demonstrate 64-channel hyperspectral imaging in the visible range with an average spectral sampling interval of 3.8 nm, achieving substantially faster reconstruction and higher fidelity than the iterative baseline. These results demonstrate that DM-FSPI provides a computationally efficient, spectrally resolved, and scalable architecture for high-channel-count single-pixel hyperspectral imaging.