Geometric Constellation Shaping Using Initialized Autoencoders

Geometric constellation shaping is a promising technique to boost the transmission capacity of communication systems. Earlier, traditional optimization methods in constellation design lead to several advanced quadrature amplitude modulation (QAM) formats, such as star QAM, cross QAM, and hexagonal QAM. The difficulty in determining decision boundaries limited their use in real systems. To overcome this, machine learning based geometric constellation shaping has recently been proposed, where the detection is done via neural networks. Unfortunately, the resulting constellation shape is often unstable and highly dependent on initialization. In this paper, we use an autoencoder for constellation shaping and detection, with strategic initialization. We contrast initialization with hexagonal QAM and square QAM. We present numerical results showing the hexagonal QAM initialization achieves the best symbol error rate performance, while the square QAM initialization has better bit error rate performance.