计算机科学
动作(物理)
人工智能
动作识别
人机交互
特征(语言学)
钥匙(锁)
视觉对象识别的认知神经科学
认知心理学
语音识别
任务分析
感知
特征提取
人工神经网络
作者
Xin Wang,Long Liu,Siying Ren,Kai Lu,Zhiyong Ma,盛鹏 Sheng Peng
标识
DOI:10.1109/tmm.2026.3726241
摘要
Skeleton-based action recognition has achieved significant progress due to the efficiency and robustness of skeletons. However, the absence of environmental descriptions limits the ability of the model to distinguish actions weakly correlated with labels. Multimodal approaches enhance accuracy by incorporating vision or language features, but often generate inaccurate or incomplete action descriptions. Moreover, multimodal features are only used as auxiliary signals during training and remain underutilized at inference. To address these issues, we propose a Multi-modality Progressive Prompt Learning (MPPL) framework to enhance the representational capacity of skeleton encoders. Specifically, we employ the Multimodal Generation (MM-Gen) module to extract informative RGB frames and action-related language descriptions from videos, enhancing environmental awareness and fine-grained details in the skeleton feature space. We design an Entropy-based Cross-Modal Alignment (ECMA) module to suppress irrelevant noise in vision and language features via entropy-weighted suppression. This achieves bidirectional cross-modal alignment and guides the learning of global skeleton representations. The Prompt Refinement Branch (PRB) module adaptively refines classification features and provides multimodal supervision during training. Specifically, PRB further exploits learnable prompts to bridge skeleton representations with category semantics, enabling a progressive guidance mechanism with ECMA for skeleton feature learning. Extensive experiments on public datasets validate the effectiveness of our approach.
科研通智能强力驱动
Strongly Powered by AbleSci AI