The performance of the emerging infrared polarization remote sensing systems is limited by the use of infrared polarization imaging sensors and cannot produce high-resolution (HR) infrared polarization images. The lack of HR infrared polarization imaging sensors and systems hinders the development and application of infrared polarization imaging technology. The existing infrared image super-resolution (SR) methods fail to improve the resolution of infrared polarization images (IRPIs) while preserving the infrared polarization information inherent in the IRPIs; thus, aiming at obtaining accurate HR infrared polarization images, this study proposed a deep-learning-based SR method, SwinIPISR, to improve infrared polarization image resolution and preserve the infrared polarization information of the target or scene. The performance of the proposed SwinIPISR was verified and compared with existing SR methods. In contrast to other methods, SwinIPISR not only improves image resolution but also retains polarization information of the scene and objects in the polarization image. Further, the impact of the network depth of SwinIPISR on the SR performance was evaluated through experiments. The experimental results confirmed the effectiveness of the proposed SwinIPISR in enhancing the image resolution and visual effects of infrared polarization images without compromising the polarization information.