EEVS: Monocular End‐Effector Pose Estimation‐Based Visual Servoing Against Uncertainties

ABSTRACT Precise robotic manipulation driven by visual perception is essential for a robot arm to autonomously execute tasks in unstructured environments. Nevertheless, uncertainties including inaccurate calibration, imperfect kinematic modeling, and dynamic environmental disturbances pose significant challenges to existing manipulation strategies. Considering that a robot arm finally performs tasks through the end‐effector, we propose a monocular end‐effector pose estimation‐based visual servoing (EEVS) method. Our method first uses a lightweight detection network to obtain a coarse pose and a mask of the end‐effector, and then refines the pose according to an optimal segmentation model. The visual servoing control of the robot arm is realized based on the relative pose between the end‐effector and the target. We theoretically demonstrate that this end‐to‐end control scheme can effectively mitigate the impact of the uncertainties. Experiments on data set demonstrate that our method achieves more accurate end‐effector pose estimation than state‐of‐the‐art approaches, especially in the scenes with significant occlusion. Moreover, real‐world robotic experiments prove that our method provides higher manipulation accuracy compared with traditional methods even in the presence of substantial kinematic errors and environmental changes.