End-to-end optimization of neural networks for orbital angular momentum-based transmission system design

We present an approach for optimizing both transmission and reception in free-space optical (FSO) communication systems employing orbital angular momentum (OAM) modes. Unlike conventional methods that rely on manually designed modulation symbol alphabets and convolutional neural networks (CNNs) for signal detection, our framework integrates an autoencoder-based optimization to jointly design the transmitter’s symbol alphabet and the receiver’s neural demodulator. By leveraging a data-driven approach, the proposed system adapts the modulation scheme to enhance detection accuracy, even under severe atmospheric turbulence. Simulation results demonstrate that this joint optimization significantly improves performance compared with prior CNN-based demodulators with fixed alphabets. The results suggest that autoencoder-based learning can be a powerful tool for dynamically optimizing OAM communication in real-world FSO environments.