Exploring augmentation strategies in mixed reality for autonomous driving with depth cameras

One significant challenge in improving autonomous driving algorithms is the lack of diverse real-world data. Moreover, transferring models from simulation to reality faces the reality gap problem. This study addresses this issue by developing an augmentation technique for mixed-reality environments, aimed at improving the testing and training of autonomous vehicles. Tested offline, it lays the groundwork for future online applications. The methodology focuses on creating virtual depth images using a virtual camera and applying an augmentation strategy to the KITTI data set. This creates a mixed-reality representation by combining virtual and real depth maps, leveraging depth information in the fusion process. The outcomes of this process are notably effective, achieving a balance between virtual and real-world aspects. This fusion method adeptly combines elements from both environments, maintaining the quality of the images. The novel contributions of this work include a detailed augmentation strategy that seamlessly integrates virtual objects into real depth maps, accounting for occlusions and ensuring realistic depth representations. In addition, this work demonstrates the feasibility of generating a large data set using the proposed method, significantly expanding the available data for training autonomous driving models. The use of metrics such as SSIM, peak signal-to-noise ratio (PSNR), and MAE, alongside object detection models such as Faster RCNN, provides a complete evaluation of both quantitative and qualitative aspects. The results demonstrate the quality of the augmented images, setting a foundation for potential online applications. The proposed strategy enables the generation of larger data set and facilitates safe, effective training in scenarios considered too risky or challenging to simulate accurately.