Learning to Analyze Human Skeletal by Radar–Camera Supervision

Skeletal detection-based analysis of human behavior is of significant value for health monitoring. This study leverages 4-D millimeter-wave (mmWave) radar technology to conduct continuous indoor skeletal analysis. Initially, we introduce the radar-visual human activity dataset (RVHAD), an extensive benchmark comprising 240000 radar frames that capture various human actions, including standing, sitting, and falling. In addition, we propose a fully automated frame correlation labeling technique capable of annotating radar frames autonomously, even in instances of visual system failure. Subsequently, we develop the spatiotemporal constrained human skeletal analysis network (STC-HSANet). This network employs a 3-D Siamese plain pyramid network (3D-SPPN) to generate multilevel collaborative features, integrates salient features through a context information interaction module (CIIM), and refines the decoded keypoint locations using a positional modulation strategy (PMS). Our experimental results demonstrate that STC-HSANet surpasses current state-of-the-art methods, offering robust performance even under conditions of visual impairment. Our code and dataset can be found at: https://github.com/zylofor/STC-HSANet.