Open-Set Domain Adaptation for Automatic Modulation Classification via Domain-Adaptive Mixup and Prototype-Based Representative Learning

Automatic modulation classification (AMC) plays a critical role in noncooperative communication and intelligent spectrum monitoring systems. Despite recent advances, two fundamental challenges remain in practical deployments. First, distribution discrepancies between training and testing environments, caused by varying channel conditions, signal-to-noise ratio (SNR) levels, and hardware impairments, often lead to severe performance degradation. Second, real-world scenarios frequently involve mismatched category spaces, where previously unseen modulation types may appear during testing, resulting in an open-set recognition (OSR) problem. To address these challenges, this article proposes a novel open-set domain adaptation (OSDA) framework for AMC that integrates domain-adaptive Mixup and prototype-based representative learning (DMPRL). In particular, a domain-adaptive Mixup module performs cross-domain interpolation between source and target samples to generate both source-dominant and target-dominant auxiliary signals, which enriches the cross-domain training distribution and improves representation robustness under domain shift. The generated samples, together with the source and target signals, are then mapped into a shared feature space through a unified feature extractor. To enhance representation robustness, a self-consistent representative learning module jointly aligns source and source-dominant samples with shared class prototypes while enforcing instance-level consistency between target and target-dominant samples. Furthermore, a collaborative representative learning module is designed to facilitate cross-domain alignment while simultaneously filtering potential unknown-class samples, thereby improving OSR capability. Extensive experiments on AMC datasets demonstrate that the proposed framework outperforms existing OSDA methods, achieving improved robustness under diverse domain shifts and unknown modulation conditions.