Semantically Guided Task Planning: Supervised Vision-Language-Action Model by Large Language Models

Enabling robots to perform everyday tasks has become increasingly important. Task planning, which decomposes task instructions into executable action sequences, is crucial for equipping robots with the ability to handle daily activities. Currently, there are two main effective methods for task planning: one relies on the reasoning capabilities of Large Language Models (LLMs), but it struggles with handling the underlying motion. The other is based on the generative capabilities of Vision-Language-Action (VLA) model, which often lacks essential semantic details. To overcome these limitations, this paper introduces a novel Semantically Supervised Vision-Language-Action (SS-VLA) model. This model addresses the constraints of previous method that relied solely on single-frame image by designing an adaptive visual sequence encoder that integrates continuous visual streams. This encoder efficiently captures and integrates multi-scale spatial and temporal features from the robot’s first-person visual perspective. Furthermore, the model utilizes LLMs to decompose task instructions into subtasks and organize them into graph structure, using Graph Attention Network (GAT) to extract features from subtask sequences and supervise the generation of action sequences. This method not only enhances the alignment of actions with task instructions but also ensures the contextual and semantic accuracy of the robot’s activities, significantly enhancing the task execution capabilities of robots in complex environments. We evaluated our model on the ALFRED and TEACh benchmark, achieving higher performance compared to existing methods, especially in unseen scenes. Additionally, we successfully deployed our model in the AI2-THOR virtual environment and on the TIAGo real robot, demonstrating the effectiveness of our method. Our code is available at: https://github.com/Li-XD-Pro/SS-VLA.