后悔
计算机科学
强化学习
透视图(图形)
结果(博弈论)
安全性令牌
度量(数据仓库)
人工智能
班级(哲学)
钥匙(锁)
比例(比率)
机器学习
块(置换群论)
数学优化
基线(sea)
相关性(法律)
分数(化学)
航程(航空)
剪裁(形态学)
汤普森抽样
会合(天文学)
考试(生物学)
采样(信号处理)
时间范围
预算约束
集合(抽象数据类型)
推论
芯(光纤)
光学(聚焦)
钢筋
元学习(计算机科学)
贴现
灵敏度(控制系统)
模式(计算机接口)
亲密度
时间预算
结构化预测
统计假设检验
任务分析
多样性(控制论)
作者
Qu Yuxiao,Yang, Matthew Y. R.,Setlur, Amrith,Tunstall, Lewis,Beeching, Edward Emanuel,Salakhutdinov, Ruslan,Kumar, Aviral
标识
DOI:10.48550/arxiv.2503.07572
摘要
Training models to effectively use test-time compute is crucial for improving the reasoning performance of LLMs. Current methods mostly do so via fine-tuning on search traces or running RL with 0/1 outcome reward, but do these approaches efficiently utilize test-time compute? Would these approaches continue to scale as the budget improves? In this paper, we try to answer these questions. We formalize the problem of optimizing test-time compute as a meta-reinforcement learning (RL) problem, which provides a principled perspective on spending test-time compute. This perspective enables us to view the long output stream from the LLM as consisting of several episodes run at test time and leads us to use a notion of cumulative regret over output tokens as a way to measure the efficacy of test-time compute. Akin to how RL algorithms can best tradeoff exploration and exploitation over training, minimizing cumulative regret would also provide the best balance between exploration and exploitation in the token stream. While we show that state-of-the-art models do not minimize regret, one can do so by maximizing a dense reward bonus in conjunction with the outcome 0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in the output stream, quantified by the change in the likelihood of eventual success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning compared to outcome-reward RL.
科研通智能强力驱动
Strongly Powered by AbleSci AI