Sleep staging using polysomnography (PSG) signals is an important method for evaluating sleep quality and aiding in the diagnosis of sleep and mental health disorders. However, manual sleep staging is a tedious and time-consuming process. Automating sleep stage classification from PSG signals is challenging due to the noisy nature of the physiological recordings and the variability of relevant features across signal types and time/frequency domains. Accordingly, we propose a hybrid network to classify sleep stages from PSG signals, i.e., electroencephalogram (EEG) and electrooculogram (EOG), along with their frequency representations. Specifically, three modules were proposed and collaboratively assembled, including the multimodal local feature extractor (MMLFE), holistic correlation encoding (HCE), and cross-domain interaction (CDI) module. For the multimodal local feature extractor (MMLFE), we employed parallel convolutional paths with different kernel sizes to capture local features of different scales in physiological signals. In the holistic correlation encoding (HCE), we integrated dilated convolution and attention mechanisms to enhance the capture capability of temporal and spectral features. The cross-domain interaction (CDI) module is further introduced to fuse multi-modal and multi-attribute features while avoiding bidirectional representation redundancy for high-quality feature mapping. We evaluate our method on the two publicly available datasets, achieving over 84% F1-score for most sleep stages. The experiment results demonstrate the ability to improve automated sleep staging compared to other SOTA methods. This has potential benefits for advancing sleep evaluation, diagnosis, and monitoring of associated health conditions.