Augmented Reality-Assisted Autonomous View Adjustment of a 6-DOF Robotic Stereo Flexible Endoscope

Stereo visual servo based surgical instruments tracking enables quicker adjustment of the view and depth of the robotic stereo flexible endoscope (RSFE). However, there are two main drawbacks: 1) when the lesion is covered by organs, the observation angles, from which the endoscope observes it, need to be adjusted independently. But this cannot be achieved by only tracking surgical instruments; 2) the field of view is restricted by the shared area of the two cameras, which limits the tracking area. Therefore, a 6-DOF augmented reality visualizing robotic stereo flexible endoscope (ARSFE), which integrates a pair of HoloLens with the RSFE, is proposed. The Hololens is used for tracking the surgeon's head and providing head-mounted display. In the ARSFE, the endoscope can be controlled in two modes: 1) The view and depth of the endoscope can be controlled by tracking surgical instruments; 2) The observation angles can be adjusted independently by tracking the surgeon's head. Modes switching is realized via the foot pedals. Besides, a hybrid control method for tracking instruments is proposed, which significantly enlarges the control field for instruments tracking. In the experiment section, the performance of the proposed hybrid visual servoing control and the ARSFE is verified.