Multi-modal Perception in Dynamic Occlusion Scenarios: A Human Pose Estimation Approach in Human-robot Collaboration

The human pose estimation technology based on robotic vision sensing systems has emerged as a critical perception approach for collision avoidance and safety assurance in human-robot collaboration (HRC). In dynamic occlusion scenarios, human pose estimation primarily relies on rapidly evolving deep learning models, yet it still faces challenges such as degraded recognition accuracy under prolonged occlusion and the high time cost of data collection and annotation. To address this issue, we proposed a novel skeletal pose and minimum-distance fusion (SPMF) approach which used a dual-RGB-D camera system to achieve robust human pose estimation under occlusion conditions. Firstly, the 3D coordinates of the joints were pre-estimated using the OpenPose-based skeletal model, in which the human body was represented by capsules at the joint positions. Following, the minimum human-robot distance was calculated via the Gilbert-Johnson-Keerthi (GJK) algorithm by combining the capsule model of the robot. In addition, a framework for human postural discrimination was established by combining the joint prediction confidence and minimum distance parameters, which can judge the reliability of pre-estimated pose data for occluded points. Finally, an algorithm for correcting misjudged joints was proposed which reconstructed the depth values for fully occluded joints. The experimental results demonstrated that the proposed human pose estimation framework achieved real-time collision detection under robot occlusion while delivering accurate pose recognition for industrial safety applications.