A robust registration method for UAV thermal infrared and visible images taken by dual-cameras

Automatic registration of unmanned aerial vehicle (UAV) thermal infrared and visible (TIR&V) images is fundamental for subsequent applications. However, few studies address this issue due to significant radiation gap, shape gap, and texture gap among TIR&V images. The area-based methods are not able to satisfy the accuracy and robustness of location at the same time, while the image pyramid-based methods are computationally expensive. To alleviate these problems, we proposed a so-called TWMM method for the registration of UAV TIR&V images taken by the camera equipped with both thermal infrared and visible sensors. TWMM is realized by combining Template matching with Weights, Multilevel local max-pooling, and Max index backtracking. TWMM consists of four steps: (1) computing similarity maps of the atomic patches using template matching with weights; (2) building pyramid similarity maps using multilevel local max-pooling; (3) deducing the corresponding points (CPs) from top to bottom using max index backtracking; and (4) eliminating outliers and estimating homography. Among the four steps, step 1 and step 2 are used to compute the similarity maps of patches with different sizes; step 3 and step 4 are used to deduce CPs and estimate homography with multiple similarity maps. TWMM was comprehensively evaluated with 600 UAV image pairs under four different scenes and also compared with current methods (i.e. SIFT, SURF, RIFT, RCB, TFeat, HardNet, RANSAC_Flow, HOPC, and CFOG). These image pairs have multiple features, i.e., different land covers, spatial resolutions, and illumination conditions, etc. Results indicate that TWMM achieves an 86.0% correct CP ratio (RCP) and a 96.0% correct matching rate (CMR) for all test images, which is a 15.1% improvement and 11.6% improvement, respectively, over the best state-of-the-art methods. TWMM also shows better robustness than other methods for weak-light images, achieving a 20.7% improvement in RCP and a 28.1% improvement in CMR. Therefore, TWMM is an effective and robust method for UAV TIR&V image registration and has good ability under different scenes.