From insight to autonomous execution: VLM-enhanced embodied agents towards digital twin-assisted human-robot collaborative assembly

• The physical entity mapping mechanism and the internal function mapping mechanism are designed to construct an embodied agent. • A VLM is constructed to be endowed with senses driven by both domain knowledge and real-time scenarios. • A digital twin model of the HRC scenario is constructed to provide a simulation and deduction engine for VLM-based HRC assembly task reasoning, which can form the embodied brain. • The complete cobot action code can be ultimately generated to accomplish the embodied execution, which can form the embodied neuron. In recent years, embodied intelligence has emerged as a practicable strategy for accomplishing human-level cognitive abilities, reasoning capacities, and execution capabilities within human-robot collaborative (HRC) assembly scenarios. As the physical instantiation of embodied intelligence, embodied agents remain largely in the exploratory phase; their practical application has yet to mature into a standardized paradigm. A key bottleneck lies in the lack of universally applicable enabling technologies, coupled with a disconnection from physical robot control systems. This deficiency necessitates repetitious training for a variety of functional models when operating in dynamic HRC environments, significantly hindering the ability of embodied agents to acclimate to complicated, dynamically changing collaborative settings. To address this challenge, this study proposes VLM-enhanced embodied agents, specifically tailored to support multimodal cognition, task reasoning, and autonomous execution in digital twin-assisted HRC assembly contexts. The framework is structured through several core steps to realize the full process closed loop from insight to autonomous execution of robots supported by embodied intelligent agents. First, a precise epsilon map relation between the embodied agent and the physical cobot is constructed, thereby enabling the digital characterization and functional capsulation of embodied agents. Building on this agent-based framework, a VLM is developed that integrates domain-specific knowledge with real-time scenario information. This dual-driven design endows the VLM with enhanced perceptual capabilities, allowing it to rapidly recognize and respond to dynamic changes in HRC scenarios. To provide a simulation and deduction engine for embodied reasoning of the assembly task, a digital twin model of the HRC scenario is built to serve as the “embodied brain”. Subsequently, these reasoning results are fed into the VLM serving as invoking parameters for the homologous sub-functional code module. This process facilitates the generation of complete robot motion code, enabling seamless physical execution and thus functioning as the “embodied neuron”. Finally, comparable experiments are conducted in an actual HRC assembly environment. The experimental results demonstrate that the proposed VLM-enhanced embodied agents have competitive advantages in multimodal cognition, task reasoning, and autonomous execution.