强化学习
计算机科学
透视图(图形)
人工智能
成对比较
机器学习
偏爱
水准点(测量)
统计模型
钥匙(锁)
偏好学习
主动学习(机器学习)
钢筋
可验证秘密共享
算法学习理论
作者
Pangpang Liu,Chengchun Shi,Will Wei Sun
标识
DOI:10.1146/annurev-statistics-042925-123430
摘要
Reinforcement learning from human feedback (RLHF) has emerged as a central framework for aligning large language models (LLMs) with human preferences. Despite its practical success, RLHF raises fundamental statistical questions because it relies on noisy, subjective, and heterogeneous feedback to learn reward models and optimize policies. This review provides a statistical perspective on RLHF, focusing primarily on LLM alignment. We introduce the main components of RLHF, including supervised fine-tuning, reward modeling, and policy optimization, and relate them to familiar statistical concepts such as the Bradley–Terry–Luce model, latent utility estimation, active learning, experimental design, and uncertainty quantification. We review methods for learning reward functions from pairwise preference data and for optimizing policies through both two-stage RLHF pipelines and emerging one-stage approaches such as direct preference optimization. We further discuss recent extensions including reinforcement learning from artificial intelligence feedback, inference-time algorithms, and reinforcement learning from verifiable rewards, as well as benchmark datasets, evaluation protocols, and open-source frameworks that support RLHF research. We conclude by highlighting open challenges in RLHF.
科研通智能强力驱动
Strongly Powered by AbleSci AI