Manifold learning algorithms for sensor fusion of image and radio-frequency data

This paper presents a joint manifold learning based heterogenous data fusion approach for image and radio frequency (RF) data. A typic scenario includes several objects (with RF emitters), which are observed by Medium Wavelength Infrared (MWIR) cameras and RF Doppler sensors. The sensor modalities of images and Doppler effects are analyzed in a way that joint manifolds can be formed by stacking up the image and Doppler data. The image data provide the aerial position and velocities of objects while the Doppler data represent the radial speeds of the objects. The proposed fusion approach exploits the manifold learning algorithms for fast and accurate sensor fusion solutions. The fusion framework has two phases: training and testing. In the training phase, the various manifold learning algorithms are applied to extract the intrinsic information via dimension reduction. Then, the raw manifold learning results (i.e., the dimension reduction results) are mapped to object trajectories of interest. The fusion results are compared with the ground truth data to evaluate the performance, based on which optimal manifold learning algorithm is selected. After the training phase, the manifold learning matrices and linear regression matrices are fixed. These matrices are used in the testing phase for multiple sensor data applications. Eight manifold learning algorithms are implemented and evaluated on Digital Imaging and Remote Sensing Image Generation (DIRSIG) scenes with MWIR data as well as distributed radiofrequency (RF) Doppler data from the same scene.