OpenVLA: An Open-Source Vision-Language-Action Model

开源 动作(物理) 计算机科学 人工智能 程序设计语言 物理 量子力学 软件
作者
Moo Jin Kim,Karl Pertsch,Siddharth Karamcheti,Ted Xiao,Ashwin Balakrishna,Suraj Nair,Rafael Rafailov,Ethan Foster,Grace Lam,Pannag Sanketi,Quan Vuong,Thomas Kollar,Benjamin Burchfiel,Russ Tedrake,Dorsa Sadigh,Sergey Levine,Percy Liang,Chelsea Finn
出处
期刊:Cornell University - arXiv [Cornell University]
被引量:40
标识
DOI:10.48550/arxiv.2406.09246
摘要

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
可爱的函函的应助被amo采纳,获得10
1秒前
1秒前
1秒前
大个的应助被zhangxu采纳,获得10
2秒前
丶蚍蜉完成签到,获得积分10
2秒前
2秒前
CodeCraft的应助被洋芋采纳,获得20
3秒前
sun12345完成签到,获得积分10
4秒前
深情安青的应助被甜美老五采纳,获得10
6秒前
6秒前
6秒前
7秒前
7秒前
wyz完成签到,获得积分10
7秒前
细腻梦凡完成签到,获得积分10
7秒前
zzzz发布了新的文献求助10
7秒前
星辰大海的应助被1077采纳,获得10
8秒前
炎炎夏无声完成签到 ,获得积分10
8秒前
虚心访枫完成签到,获得积分10
8秒前
9秒前
9秒前
9秒前
10秒前
科目三的应助被慧子采纳,获得30
10秒前
11秒前
ding的应助被鲤鱼鑫磊采纳,获得10
12秒前
13秒前
执着发布了新的文献求助10
13秒前
JamesPei的应助被学术饕餮采纳,获得10
13秒前
英姑的应助被duzhuo采纳,获得10
14秒前
zzc发布了新的文献求助10
14秒前
领导范儿的应助被从心采纳,获得10
15秒前
风继续吹完成签到,获得积分10
15秒前
wyz发布了新的文献求助10
15秒前
16秒前
16秒前
心如止水完成签到,获得积分10
16秒前
纸箱发布了新的文献求助10
16秒前
湖蓝色块完成签到 ,获得积分10
17秒前
大胆初南完成签到 ,获得积分10
18秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1700
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
八旗制度の研究 600
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8187835
求助须知:如何正确求助?哪些是违规求助? 9575825
关于积分的说明 21843558
捐赠科研通 7785715
什么是DOI,文献DOI怎么找? 3435125
关于科研通互助平台的介绍 2559495
邀请新用户注册赠送积分活动 2482008