Cooperative pursuit of underwater dynamic targets: a heterogeneous graph attention network-based multi-agent reinforcement learning approach

As marine intelligent equipment technology advances, leveraging heterogeneous systems composed of Autonomous Underwater Vehicles (AUVs) and Unmanned Surface Vehicles (USVs) for cooperative operations has emerged as a frontier in addressing complex underwater challenges. In specific scenarios such as 3D dynamic target pursuit, overcoming the inherent limitations of underwater communication and perception is key to enhancing overall system performance. Addressing the common challenges in existing reinforcement learning methods regarding communication efficiency, modeling heterogeneous interactions, and engineering-oriented control under realistic dynamics, this paper proposes a Heterogeneous Graph Attention Network-based Heterogeneous Multi-Agent Soft Actor-Critic (HGA-HASAC) framework. This framework first designs a Heterogeneous Relational Abstraction Network (HeRANet), which utilizes hard and soft attention mechanisms to achieve selective aggregation of neighbor information based on spatial distribution and agent type, thereby effectively capturing complex interactions and reducing communication overhead. Secondly, a heterogeneous edge-enhanced graph attention module is introduced into the critic network to accurately evaluate the joint policy value of the heterogeneous system. Furthermore, this paper abandons simplified kinematic models and constructs an end-to-end control strategy based on the true dynamic equations of AUVs and USVs. This strategy directly outputs low-level control commands to the actuators, thereby improving the practical fidelity and engineering relevance of the learned policies. Numerical simulation results demonstrate that, compared to baseline algorithms, the proposed HGA-HASAC framework shows significant advantages in both pursuit success rate and cooperative efficiency, validating its effectiveness and superiority in solving the cooperative pursuit problem for heterogeneous marine robotic systems.