Multimodal Perception for Indoor Mobile Robotics Navigation and Safe Manipulation

Indoor mobile robotics (IMR) has gained significant attention due to its potential applications in various domains, such as healthcare, logistics, and domestic assistance. However, navigating through indoor environments and performing safe manipulations still pose intractable challenges in terms of navigation accuracy and obstacle avoidance. To solve these issues, this article presents an artificial intelligence (AI) embodied multimodal perception framework for IMR intelligent navigation and safe manipulation. To ensure the navigation accuracy and robustness, we employ the complementary forward RGB camera, downward QR vision sensor, and wheel encoder measurements in a unified framework. The visual residuals and wheel odometry residuals are jointly minimized to estimate the robot states. To guarantee the safety of robotic manipulation tasks, we have developed an AI model that integrates transformer network with convolutional neural network, to associate the long-range RGB & depth patches and aggregate the multiscale obstacle features, enabling the precise detection and segmentation of obstacles in RGB-D images. Afterwards, the depths of detected obstacles are regressed, providing the robot with crucial information for collision avoidance. Eventually, we design a refined robot manipulation system that dynamically adjusts the robot behavior to ensure effective collision avoidance and to minimize potential damage to its mechanical components by constantly evaluating the spatial relationships between the robot and its surroundings. By incorporating advanced obstacle detection and the avoidance mechanism, mobile robots can navigate reliably in indoor environments with a reduced risk of collisions and real-time decision making. The presented method has been evaluated on the developed IMR platform. On the collected dataset, the estimated IMR absolute position and orientation errors are less than 0.18 m and 5${}^{\boldsymbol{\circ}}$, respectively. Besides, it achieves 89% $mAP$ on obstacle detection. The maximum of the estimated obstacle relative depth & orientation errors are less than 0.4 m and 2${}^{\boldsymbol{\circ}}$, respectively, which proves competitiveness against the state-of-the-art in both robot navigation and safe manipulation.