Deep Transfer Reinforcement Learning for Beamforming and Resource Allocation in Multi-Cell MISO-OFDMA Systems

Orthogonal frequency division multiple access (OFDMA) is one of the promising technologies to satisfy the huge access demand and high data-rate requirement of the fifth generation (5G) networks. In this paper, we study the joint beamforming coordination and resource allocation in the downlink multi-cell multiple-input single-output OFDMA (MISO-OFDMA) systems. First, we divide the allocation framework into beamforming coordination and power allocation (BCPA) module and subcarrier allocation (SA) module. Then, we design a multi-agent deep Q-network (MADQN) algorithm for the allocation framework. Furthermore, we propose a MADQN-based transfer learning framework using knowledge distillation, which is called transfer learning-MADQN (TL-MADQN), to improve the adaptability of neural networks for different wireless schemes. TL-MADQN exploits neural networks and their parameters distilled from pre-trained agents and the experience collected from new agents so that the new agents complete their training process effectively and quickly in the new network environment. Finally, we adjust the allocation policy to maximize the sum data-rate for all users by updating the weights of each neural network. Simulation results show that the proposed MADQN algorithm achieves better performance than the baseline algorithms. Moreover, our TL-MADQN framework further improves the convergence speed and data-rate, which validates its effectiveness and superiority.