Contrastive Feedback Vision-Language for 3D Skeleton-Based Action Recognition

Large contrastive vision-language models (VLMs) have recently shown promise in skeleton-based action recognition. However, given the lack of skeleton frame-text training datasets for VLMs, aligning the representations between the skeleton frames and labels remains challenging. Specifically, two key limitations must be addressed. First, VLMs struggle to align abstract action labels' language representations with sequential skeleton frames containing primary action semantics, impeding the ability of language representations to represent primary action information effectively. Second, vision representations with high-order action information are difficult to align with labels' language representations because of the risk of homogenizing discriminative features from different data streams. To address these challenges, we propose a Contrastive Feedback Vision-Language (CFVL) model for 3D skeleton-based action recognition that consists of a language representations' feedback decoder and a data stream-adaptive projection module. The feedback decoder aligns the decoded language representations with the original skeleton inputs to help the model comprehend primary action vision information. The projection module employs adaptive structures to further extract spatiotemporal information from various data streams. Additionally, the data stream-adaptive projection module projects vision and text language representations into a unified high-latency semantic space. Discriminative action vision representations, along with consistent representation spaces, support the effective alignment of vision-language representations with high-order action information. The experimental results demonstrate the superior performance of the proposed CFVL model on the Northwestern-UCLA, PKU MMD, NTU RGB+D 60/120, and FSD-10 datasets.