计算机科学
强化学习
管道(软件)
分类学(生物学)
动作(物理)
自然语言
数据科学
人工智能
自然语言理解
钥匙(锁)
语言模型
人机交互
开放式研究
知识管理
管理科学
弹道
作者
Qianyue Hao,Lin Chen,Xin-Kun Qi,Yuan Yuan,Zefang Zong,H. S. Chen,Keyu Zhao,Shengyuan Wang,Yunke Zhang,Jian Yuan,Yong Li
摘要
Reinforcement learning (RL), is becoming essential in the post-training of large language models (LLMs), enhancing their capabilities and alignment with human preferences. However, adapting conventional RL to LLMs introduces challenges stemming from their massive parameter size and the vast natural language action space. In this survey, we conduct a systematic literature review on how RL are adapted and scaled as a fundamental post-training tools. First, we provide a taxonomy of challenges faced in each stage of the RL training loop, including action exploration, trajectory collection, reward evaluation, and model update. We then introduce recently developed methods to address these issues, ranging from logical structure navigation, training data curation to reward design and advantage estimation. Afterward, we elaborate on the application of these techniques in diverse domains such as mathematics, coding, medicine, and information retrieval, analyzing how innovations in the RL pipeline enhance the domain-specific LLMs. Finally, we discuss the limitations and side effects of applying RL to LLMs and explore open problems and future directions like the balance between efficiency and effectiveness, and algorithm-system co-design. This survey helps researchers understand recent progress and inspire novel research to address current challenges and realize the full potential of RL for LLMs.
科研通智能强力驱动
Strongly Powered by AbleSci AI