Abstract With the rapid development of the internet of vehicles (IoV) system and intelligent applications, vehicles are generating increasingly computation-intensive tasks, which impose stringent demands on real-time performance, energy efficiency, and resource management. Recently, multi-agent deep reinforcement learning (MADRL) has emerged as a promising solution for task offloading in such dynamic environments, owing to its adaptability and low decision-making complexity. However, most existing approaches predominantly focus on resource constraints at the vehicle level, often neglecting the limited computing and storage capacities of edge servers, such as roadside units (RSUs), which results in inefficient offloading and coordination decisions. To address these challenges, we propose the Combinatorial Multi-Agent Actor-Critic with Coordination (COMA2C) algorithm. In this algorithm, vehicle agents dynamically generate task offloading requests based on link conditions and local resource availability, while RSU agents, constrained by limited resources, perform centralized decision-making through a combinatorial action selection mechanism. This enables global resource coordination and efficient task allocation. Simulation results under typical Vehicular Edge Computing (VEC) scenarios show that the proposed approach outperforms existing benchmarks in terms of task completion rate, task delay and energy consumption, thereby validating its effectiveness and scalability in real-world vehicular environments.