This paper describes an automatic sleep staging network based on R-peak-to-R-peak intervals (RRI) which was designed to make sleep state classification without manual feature designing. This network consists of two parts, the first part completed feature extraction and fusion through convolutional neural network, and the sequence dependencies was extracted by second part through temporal convolutional network (TCN). To validate this model, we used sleep data of 500 people in the Sleep Heart Health Study database for training and testing. On the test data of 100 people, the network can obtain an accuracy of 83.3% and a kappa coefficient of 0.71 on the three-stage classification task (Wake, NREM, REM). Further, considering the physiological information contained in the respiration during sleep, we also adding the breath-to-breath intervals as the input, an accuracy of 86.5% and a kappa coefficient of 0.77 was obtained. Compared with the reproduce results of related studies in recent years, the model is more accurate and more consistent with the labels diagnosed by physicians.