Towards human-centric smart manufacturing : a systematic vision-language model-based methodology for human-robot interactive, task planning, navigation, and manipulation

作者
Tian Wang
出处
摘要

Rooted in the principles of Industry 5.0, Human-centric Smart Manufacturing (HSM) prioritizes human welfare within manufacturing processes, leveraging Artificial Intelligence (AI) to simultaneously improve operational efficiency and employee well-being. Under this paradigm, Human-Robot Interaction (HRI) technologies are central because they unify human problem-solving and decision-making strengths with the precision and efficiency of robotic systems, resulting in adaptive, intelligent, and superior manufacturing processes. Nevertheless, existing HRI works commonly face several challenges: 1) Data scarcity for specific tasks - limited availability of data for particular tasks can hinder the development of efficient models or systems; 2) Requirement for re-training for different scenarios - many existing approaches necessitate re-training or fine-tuning models for different scenarios, which can be time-consuming and impractical; 3) Predominantly pre-defined manners - most interaction approaches are predetermined, offering limited adaptability and failing to support open-ended or natural exchanges between humans and robots.. Recently, Vision-Language Models (VLMs) and Large Language Models (LLMs) have attracted significant attention in the field of HRI, and they hold strong potential as effective solutions to address the current limitations faced in HRI. Firstly, as pretrained on vast repositories of world knowledge, VLMs and LLMs possess the capacity to generalize across diverse scenarios and tasks, thereby addressing the limitations of previous models that suffered from insufficient datasets and required frequent retraining, thus rendering HRI more adaptable and effective across multiple domains. Secondly, VLMs strengthen robotic intelligence by fusing visual and linguistic inputs with unified reasoning, thereby mirroring the multimodal perception characteristic of humans. Thirdly, VLMs and LLMs enhance user experience by enabling flexible interaction and communication, thereby making HRI more natural, efficient, and intuitive. In this case, this research aims to develop a VLM-based HRI methodology that makes it possible for robots to engage with humans naturally and systematically follow their commands to collaborate on accomplishing complex tasks—such as supplementing assembly materials, repairing malfunctioning machines, and detecting potential safety issues—toward future human-centric smart manufacturing. To realize this goal, three fundamental robotic tasks warrant exploration: robot system task planning, mobile robot navigation, and robot arm manipulation. Accordingly, this research proposes a systematic vision-language model-based methodology for human-robot interactive task planning, navigation, and manipulation in manufacturing. Chapter 2 provides a survey of vision-language techniques applied to HRI within human-centered smart manufacturing, and concludes by outlining unresolved challenges in task planning, navigation, and manipulation. In Chapter 3, the background and research challenge of vision and language-based task planning are introduced, and a coarse-to-fine vision-grounded Chain-of-Thought (CoT) reasoning approach is proposed to address the unforeseen challenge faced by smart manufacturing systems. By integrating multi-granularity visual parsing with a coarse-to-fine CoT framework, our method jointly leverages global scene context and local visual perspectives to enable interpretable, zero-shot reasoning, while generating clear and precise robot-executable task plans. The evaluation results indicate that our framework attains remarkable success and generalization in challenging as well as novel scenarios, emphasizing its reliability and applicability to real-world smart manufacturing environments. In Chapter 4, the background and research gaps of vision and language navigation is introduced. Then proposes a human-guided mobile robot navigation method for unstructured manufacturing environment. Finally, we use some experiments to evaluate these methods. Learning-based VLN methods heavily rely on data and are constrained by the availability of a fixed number of scenes for training. This limitation prevents them from effectively addressing the challenges of realistic application scenarios, where objects may exist outside the pre-defined domains and language instructions can be more diverse. Additionally, although the current method shows promising performance in simulated environments, it faces challenges when applied in real-world scenarios due to sensor noise. This noise can lead to the loss of geometric details in the reconstructed maps, resulting in inaccurate segmentation of the navigation target. To tackle these issues, we integrate classical 3D reconstruction techniques with zero-shot VLM, aiming to construct an open-vocabulary 3D semantic mapping framework. Furthermore, an LLM is utilized to interpret natural language commands and produce control code for robots, thereby facilitating navigation supported by vision and language. In Chapter 5, the background and research challenge of vision and language-based robot manipulation is firstly introduced, and then a foundational model-based framework is proposed for 3D Hierarchical Affordance Reasoning, concentrating on high-precision tool operations in factory environments. Within this framework, two components are proposed: 3D Joint Tool-Part Affordance Grounding enables open-vocabulary 3D affordance detection, jointly grounding tool and part interactions; task-aware Trajectory Generation leverages an LLM alongside motion affordance priors to generate task-specific motion trajectories, enabling robots to perform complex tool-use operations with high precision and adaptability. Extensive experiments show that the proposed hierarchical affordance reasoning framework demonstrates substantial improvements over current leading approaches. Experiments carried out on actual manufacturing systems confirm the effectiveness and stability of the proposed approach, highlighting its suitability for real-world industrial settings that require strong dexterity, logical reasoning, and generalization capabilities. It is hoped that this work can create a multi-robot intelligent system capable of achieving human-centric natural visual-language interaction. Under human guidance, this system will be able to accomplish various tasks in complex factory environments in future human-robot symbiotic manufacturing paradigms.

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
飞儿完成签到 ,获得积分10
1秒前
369ninja发布了新的文献求助10
1秒前
兴奋的惮完成签到,获得积分10
2秒前
进击的牛马完成签到,获得积分10
2秒前
游01完成签到 ,获得积分10
4秒前
打打的应助被活泼雪莲采纳,获得10
6秒前
zhuangbaobao完成签到,获得积分10
6秒前
7秒前
个性怡完成签到 ,获得积分10
7秒前
深情的乐萱完成签到,获得积分10
7秒前
雪山大地发布了新的文献求助10
9秒前
失眠的寄云完成签到,获得积分10
12秒前
12秒前
O_0完成签到 ,获得积分10
12秒前
WQ完成签到,获得积分10
12秒前
莹莹大王发布了新的文献求助10
17秒前
落月无痕完成签到,获得积分10
19秒前
win050532完成签到 ,获得积分10
20秒前
吃饭打肯德基完成签到 ,获得积分10
22秒前
马伯乐完成签到 ,获得积分10
24秒前
Qinzhiyuan1990完成签到 ,获得积分10
24秒前
Lytoo完成签到 ,获得积分10
25秒前
26秒前
Enyu完成签到 ,获得积分10
28秒前
紧张的绮玉完成签到 ,获得积分10
31秒前
雪山大地发布了新的文献求助10
31秒前
32秒前
喂喂喂完成签到 ,获得积分10
35秒前
Bismarck完成签到 ,获得积分10
35秒前
wanci的应助被酷酷的怀莲采纳,获得50
37秒前
活泼雪莲发布了新的文献求助10
38秒前
41秒前
爱而不得完成签到 ,获得积分10
41秒前
科研摆渡人完成签到,获得积分10
42秒前
laber完成签到,获得积分0
42秒前
wang完成签到 ,获得积分10
45秒前
xipding完成签到,获得积分10
45秒前
47秒前
huluwa完成签到,获得积分10
51秒前
yihao完成签到 ,获得积分10
52秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1700
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
John McDowells Bildungstheorie 800
八旗制度の研究 600
Hidden Empire of Finance: How Wall Street Profits from Our Cities and Fuels Global Inequality 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8166550
求助须知:如何正确求助?哪些是违规求助? 9559535
关于积分的说明 21750720
捐赠科研通 7757357
什么是DOI,文献DOI怎么找? 3429141
关于科研通互助平台的介绍 2553523
邀请新用户注册赠送积分活动 2473904