Joint Tx and Rx Look-Up-Table Based Nonlinear Distortion Mitigation in Reduced State MLSE for 180 Gbit/s PAM-8 IM-DD System

Inter-symbol-interference (ISI) and nonlinear distortions are the major impairments in high-speed short-reach intensity modulation and direct detection (IM-DD) systems. Linear ISI can be well addressed with linear equalizer cascaded maximum likelihood sequence estimation (MLSE), while the nonlinear distortions cannot be effectively compensated in this manner. In this article, a low complexity nonlinear compensation scheme jointing Tx look-up-table (LUT) and Rx LUT based nonlinear MLSE is proposed to fully eliminate the system nonlinear distortions. The effectiveness of the proposed scheme is experimentally verified in 150/165/180 Gbit/s PAM8 systems over 20-km single-mode fiber transmission at O-band. The experimental results of the 150/165 Gbit/s PAM-8 system demonstrate that the proposed scheme exhibits 1∼2 dB receiver sensitivity improvement compared to the scheme only with one LUT on the transmitter or receiver at the bit error rate (BER) of 7% hard-decision forward error correction (HD-FEC) threshold of 3.8 × 10 ⁻³ . In addition, the proposed scheme can support the transmission of 180 Gbit/s PAM-8 signal over 20-km SSMF with the BER below the 7% HD-FEC threshold. Finally, a nonlinear reduced state MLSE (RS-MLSE) based on joint Tx and Rx LUT is also investigated to reduce the complexity of the MLSE decoder. The results show that it can reduce 98.44% multiplications, 98.44% additions, and 84.38% comparisons while with a negligible BER performance penalty.