Knowledge-Guided Multi-Agent Reinforcement Learning for Multi-AAV Cooperative Pursuit With Sparse Team Rewards

With the rapid development of urban air mobility, cooperative pursuit with multiple autonomous aerial vehicles (multi-AAV) has become increasingly important for airspace security and emergency response applications. Multi-agent reinforcement learning (MARL) has emerged as a mainstream paradigm for this problem. However, existing MARL methods mainly rely on dense individual rewards that encourage agents to maximize personal gains rather than team objectives, leading to suboptimal coordination. Consequently, learning effective cooperative pursuit policies under sparse team rewards remains a significant challenge, especially in obstacle-dense environments. In this article, we propose KG-MASAC, a knowledge-guided MARL approach for multi-AAV cooperative pursuit. To address this challenge, we introduce two knowledge guidance mechanisms. First, knowledge-augmented policy optimization provides coordination guidance through a cooperative force field. Second, an adaptive reward transition guides early exploration with dense auxiliary rewards, then progressively shifts to sparse team rewards as cooperative performance improves. We validate KG-MASAC through extensive simulations and physical experiments on indoor and outdoor AAV platforms. Results show that KG-MASAC achieves superior capture success, robust generalization to unseen obstacle densities and team sizes, emergent cooperative behaviors such as herding and encirclement, and successful sim-to-real transfer under environmental uncertainties. Note to Practitioners—Multi-AAV cooperative pursuit of fast-moving targets in obstacle-dense environments underpins applications from low-altitude airspace patrol to intruder interception. However, existing methods rely on hand-shaped individual rewards that demand extensive tuning but often produce selfish rather than cooperative behaviors. To address this, we present KG-MASAC, a practical approach that learns team-oriented pursuit policies from sparse task-level feedback. By leveraging force field priors and an adaptive reward curriculum that automatically transitions from dense to sparse signals, KG-MASAC reduces the need for manual reward shaping and curriculum design. Once trained, the policy transfers zero-shot to both indoor motion-capture arenas and outdoor GPS-based settings, demonstrating robust generalization across quadrotor platforms, localization modalities, and environmental disturbances. Future work will explore dynamic obstacles and multi-target pursuit to further extend this generalization.