Trajectory Design and Beamforming in UAV-Assisted Wireless Networks: A Fine-Tuned M2LLM-Driven DRL-Based Framework

Optimizing unmanned aerial vehicle (UAV)-assisted wireless networks to serve mobile users (MUs) via beamforming presents significant challenges, mainly due to the dynamic and complex environments. Traditional single-modal data-based modeling methods are often insufficient for capturing the varying environmental characteristics, leading to inaccurate UAV trajectory design and beamforming. To address these issues, we propose a multi-UAV-assisted integrated sensing, communication, and computation (ISCC) framework that processes multi-modal data to enhance environmental awareness and improve communication performance. We then formulate an optimization problem to maximize the average sum rate by jointly optimizing the UAV trajectory and beamforming vectors. Given the non-convex nature of the problem, traditional optimization techniques are inadequate. To this end, we introduce a fine-tuned multi-modal large language model (M2LLM)-driven deep reinforcement learning (DRL)-based joint optimization framework. Specifically, a pre-trained M2LLM is first fine-tuned to predict future MU positions by leveraging historical multi-modal data, including texts, images, and wireless sensing data. The fine-tuned M2LLM is then employed to extract environmental features, where the output of the fine-tuned M2LLM’s last hidden layer is regarded as the environment state vector to eliminate the output uncertainty of the M2LLM. Subsequently, we use a DRL agent to optimize the UAV trajectory and beamforming in a coordinated manner. Extensive simulation results demonstrate that the proposed framework can significantly enhance network performance by enabling environment-aware and adaptive trajectory design and beamforming. The code is available in https://huggingface.co/blYin/MmllmDrlUavTdBf.