Optimal Constellation Design to Combat Equalization-Enhanced Phase Noise

In fiber communication systems employing coherent receivers, local oscillator laser phase noise should ideally be compensated prior to dispersion equalization. However, practical limitations in current receiver implementations necessitate reversing this order, such that phase noise compensation is performed after dispersion equalization. This processing sequence amplifies the phase noise, resulting in equalization-enhanced phase noise (EEPN), which has emerged as a critical performance-limiting factor, particularly in high-baud-rate or long-reach transmission systems with significant accumulated dispersion. In this work, recognizing that EEPN-induced impairment is predominantly governed by the phase noise component, we propose a novel distance metric for symbol detection and constellation design that departs from the conventional Euclidean distance. Based on this metric, we optimize the constellation geometry to enhance system robustness against EEPN. Simulation results demonstrate that the optimized constellations achieve lower symbol error rates over extended transmission distances compared to conventional designs. Furthermore, when EEPN is the dominant impairment, the proposed approach enables a 10% increase in baud rate at an equivalent symbol error rate relative to state-of-the-art shaped constellations.