Interactive Construction Robots for Timber Truss Assembly via Vision-Language Model and Mark-Based Visual Prompting

Intelligent construction robots are deemed the future of on-site construction for improved productivity and safety. Despite recent developments in learning-based control methods, effective human–robot interaction remains challenging for construction tasks. To fill this gap, we propose a control system for on-site construction robots that converts instructions of human workers to executable robot actions, with an option for human intervention. Our system utilizes the segment anything model and keypoint sampling to extract point representations on the timber frame and employs vision-language model and mark-based visual prompting to translate instructions and point representations into actions. To evaluate the efficacy of our system, we test it on a UR10 robot arm in simulation, focusing on the timber truss assembly task, where a robot arm is tasked with timber placement within a timber frame. In experiments, our system achieved a success rate of 75% in 20 trials, with detailed analysis identifying reasons for failures.