Boundary‐Aware Learning for Robust Automatic Modulation Recognition Based on Adaptive Pyramid Network

ABSTRACT Automatic modulation recognition (AMR) of radar signals plays a pivotal role in intelligent spectrum management for modern electronic warfare systems. Although deep learning has improved AMR accuracy, existing approaches encounter challenges when identifying signal types beyond those seen during training. To overcome this limitation, this work proposes a boundary‐aware learning (BAL) framework with three innovations. First, an adaptive pyramid network automatically highlights important time‐frequency information at multiple dimensions, producing richer and more discriminative signal representations. Second, a boundary‐aware learning strategy shapes the embedding space so that samples of the same modulation are drawn tightly together, whereas different modulations are pushed farther apart. Third, a distance‐based rejection mechanism measures how far a new signal lies from known clusters, enabling reliable detection of previously unseen modulations. Together, these components create a unified feature space that both sharpens class boundaries and isolates unknown signal types. Extensive experiments on both simulated and measured radar datasets show that the proposed framework outperforms conventional architectures in open‐set signal recognition, confirming its superior robustness in realistic electromagnetic environments.