In hyperspectral images, ghost image residuals exceeding a certain threshold not only reduce the recognition accuracy of the imaging detection system but also decrease the target identification rate. Ghost image residuals affect both the recognition accuracy of the detection system and the accuracy of spectral calibration, thereby influencing qualitative and quantitative inversion. Conventional ghost image residual correction methods can significantly affect both the relative and absolute calibration accuracy of hyperspectral images. To minimize the impact on spectral calibration accuracy during ghost image residual correction, we propose a ghost image degradation model and an iterative optimization algorithm. In the proposed approach, a ghost image residual degradation model is constructed based on the point spread function (PSF) of ghost image residuals and their energy distribution characteristics. Using the proportion of ghost image residuals and the accuracy of hyperspectral image calibration as constraints, we iteratively optimized typical regional target ghost image residuals across different spectral channels, achieving automated correction of ghost image residuals in various spectral bands. The experimental results show that the energy proportion of ghost image residuals at different wavelengths decreased from 4.6% to 0.3%, the variations in spectral curves before and after correction were less than 0.8%, and the change in absolute radiometric calibration accuracy was below 0.06%.