Robotic manipulation based on 3-D visual servoing and deep neural networks

The critical challenge, for robot–object-interaction, is to estimate visually the pose of the target object in a 3D space and combine it into a vision-based control scheme in manipulation applications. This paper proposes a novel reliable framework for deep ConvNet combined with visual servoing using a single RGB camera. We introduce an extensive system called Deep-Visual-Servoing (DVS) that addresses an integration of: (I) training of deep-CNNs using synthetic dataset only and operates successfully in real-world scenario, (II) continuous 3 D object pose estimation as the sensing feedback in a 3D visual servoing control scheme, and (III) design, integration and experimentation of visual servoing approach based on Lyapunov’s theory. The proposed deep based learning approach, the kinematic modeling and controller design are experimentally verified and discussed using the 6 DOF UR5 manipulator.