Multiagent Reinforcement Learning: Rollout and Policy Iteration

稳健性(进化) 计算机科学 强化学习 计算 基础(拓扑) 数学优化 状态空间 国家(计算机科学) 数学 算法 人工智能 数学分析 生物化学 化学 统计 基因
作者
Dimitri P. Bertsekas
出处
期刊:IEEE/CAA Journal of Automatica Sinica [Institute of Electrical and Electronics Engineers]
卷期号:8 (2): 249-272 被引量:138
标识
DOI:10.1109/jas.2021.1003814
摘要

We discuss the solution of complex multistage decision problems using methods that are based on the idea of policy iteration (PI), i.e., start from some base policy and generate an improved policy. Rollout is the simplest method of this type, where just one improved policy is generated. We can view PI as repeated application of rollout, where the rollout policy at each iteration serves as the base policy for the next iteration. In contrast with PI, rollout has a robustness property: it can be applied on-line and is suitable for on-line replanning. Moreover, rollout can use as base policy one of the policies produced by PI, thereby improving on that policy. This is the type of scheme underlying the prominently successful AlphaZero chess program. In this paper we focus on rollout and PI-like methods for problems where the control consists of multiple components each selected (conceptually) by a separate agent. This is the class of multiagent problems where the agents have a shared objective function, and a shared and perfect state information. Based on a problem reformulation that trades off control space complexity with state space complexity, we develop an approach, whereby at every stage, the agents sequentially (one-at-a-time) execute a local rollout algorithm that uses a base policy, together with some coordinating information from the other agents. The amount of total computation required at every stage grows linearly with the number of agents. By contrast, in the standard rollout algorithm, the amount of total computation grows exponentially with the number of agents. Despite the dramatic reduction in required computation, we show that our multiagent rollout algorithm has the fundamental cost improvement property of standard rollout: it guarantees an improved performance relative to the base policy. We also discuss autonomous multiagent rollout schemes that allow the agents to make decisions autonomously through the use of precomputed signaling information, which is sufficient to maintain the cost improvement property, without any on-line coordination of control selection between the agents. For discounted and other infinite horizon problems, we also consider exact and approximate PI algorithms involving a new type of one-agent-at-a-time policy improvement operation. For one of our PI algorithms, we prove convergence to an agent-by-agent optimal policy, thus establishing a connection with the theory of teams. For another PI algorithm, which is executed over a more complex state space, we prove convergence to an optimal policy. Approximate forms of these algorithms are also given, based on the use of policy and value neural networks. These PI algorithms, in both their exact and their approximate form are strictly off-line methods, but they can be used to provide a base policy for use in an on-line multiagent rollout scheme.

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
lei完成签到,获得积分10
刚刚
zhaoxi完成签到 ,获得积分10
1秒前
1秒前
丘比特的应助被zyf采纳,获得10
2秒前
jojo完成签到,获得积分10
2秒前
2秒前
kata完成签到,获得积分10
2秒前
南风完成签到,获得积分10
3秒前
夜寻发布了新的文献求助30
4秒前
无花果的应助被meng采纳,获得10
4秒前
zhikaiyici发布了新的文献求助30
5秒前
刘昊然的老婆完成签到 ,获得积分10
6秒前
liu发布了新的文献求助10
6秒前
Zhao完成签到 ,获得积分10
6秒前
Felix发布了新的文献求助10
7秒前
小蘑菇的应助被xiaoxiao采纳,获得10
8秒前
bad boy发布了新的文献求助10
8秒前
天天快乐的应助被李李李采纳,获得10
8秒前
9秒前
9秒前
小小鱼完成签到,获得积分10
9秒前
英俊的铭的应助被猪大肠采纳,获得10
10秒前
11秒前
11秒前
酷波er的应助被玩家666采纳,获得10
11秒前
11秒前
SciGPT的应助被夜寻采纳,获得30
12秒前
星辰大海的应助被魏同心采纳,获得10
12秒前
打打的应助被arcidia采纳,获得10
13秒前
14秒前
15秒前
MMLi发布了新的文献求助10
15秒前
zyf发布了新的文献求助10
15秒前
FashionBoy的应助被Jasmine采纳,获得10
15秒前
15秒前
16秒前
16秒前
毛毛发布了新的文献求助10
16秒前
maguodrgon发布了新的文献求助10
16秒前
深情可乐完成签到,获得积分10
16秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 2000
Handbook of Sport Coaching Research 1000
Orthognathic Surgery - 2 Volume Set 500
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
基于结构调控的功能化水诱导发电材料与器件研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8193653
求助须知:如何正确求助?哪些是违规求助? 9579816
关于积分的说明 21867585
捐赠科研通 7793288
什么是DOI,文献DOI怎么找? 3437343
关于科研通互助平台的介绍 2560779
邀请新用户注册赠送积分活动 2484193