Predictive display for teleoperation with virtual reality fusion technology

Abstract A virtual reality (VR) fusion technology of matching a virtual environment of simulated graphics models with actual camera views of the slave station task environment has been developed. This fusion technology enables high‐fidelity predictive displays with calibrated graphics overlay on the live video. Reliable and accurate parameters are achieved by camera calibration and feature point detection. Since the object pose becomes known through the virtual model, the operator can use the teleoperation system effectively. Predictive displays are very useful for both noncontact and contact tasks, providing an effective VR interface with immediate visual prediction to the operator. Besides, the feature points detection avoids the exacerbation of maneuverability caused by outside interference. Lastly, the experiment of endoscopic penetration is conducted to validate the system.