Open-Set Automatic Modulation Recognition Based on Circular Prototype Learning and Denoising Diffusion Model

Automatic modulation recognition (AMR) technology is a critical component of modern communication systems. However, conventional AMR methods based on the closed-set assumption struggle to detect unknown classes that may appear during testing. To address this limitation, this paper proposes an open-set automatic modulation recognition (OSAMR) framework, termed CPLDiff, that integrates circular prototype learning (CPL) with a denoising diffusion model (DDM) to detect unknown classes. The core idea of CPLDiff is to jointly leverage the class-level and instance-level information of the training samples. To achieve this, CPL is used to extract class-level information, while the diffusion model is employed to extract instance-level information. (1) Circular Prototype Learning: Prototype vectors are pre-optimized and fixed, and a bias radius is introduced to expand the feasible encoding space. (2) Denoising Diffusion Model: Noise is added to the sample, and the DDM is used to remove this noise. The probability of a sample belonging to a known class is proportional to the extent of noise removal. (3) Final Integration: The outputs of the CPL and the DDM are combined to perform OSAMR. We conducted comparative experiments and evaluated the proposed method using diverse metrics to ensure a comprehensive assessment of its effectiveness. The experimental results demonstrate that the CPLDiff method significantly improves the detection capability for unknown classes compared to state-of-the-art methods.