Significant wave height (SWH) plays a critical role in marine operations, ship navigation, and climate prediction. The X-band radar is widely used for SWH inversion due to its short wavelength and high attenuation rate. Current inversion methods mainly fall into two categories: spatiotemporal-domain methods based on deep learning, and frequency-domain methods based on physical models. However, traditional deep learning methods typically emphasize spatial features while neglecting sequential information in radar image processing, which results in the loss of temporal features. Additionally, physics-based inversion methods in frequency-domain rely on manual feature design and cannot adaptively learn the nonlinear mapping between frequency feature and SWH, which hinders the improvement of inversion accuracy. To solve these problems, this paper proposes a bimodal cross-attention spatiotemporal-frequency fusion inversion network (BiCross-STFNet) to estimate SWH. First, the radar image sequences are simultaneously input into the spatiotemporal feature extraction module and frequency feature extraction module, where spatiotemporal and frequency features are extracted using the 3D spatiotemporal aware residual block. These features are then aligned and fused through the bimodal cross-attention fusion module, and finally, SWH is estimated. Experimental results demonstrate that BiCross-STFNet achieves a correlation coefficient of 0.964, a root mean square deviation (RMSD) as low as 0.04, and a mean absolute percentage error (MAPE) of 5.59%, outperforming existing methods.