In automatic modulation recognition (AMR) tasks, the performance of deep learning-based automatic modulation classification models degrades due to feature distribution shifts caused by the superposition of multiple factors. Therefore, this paper proposes a lightweight domain adaptive modulation recognition method (DAAMR) based on dual-classifier adversarial learning. The method constructs an adversarial framework consisting of a lightweight multi-branch feature extractor and dual classifiers, and achieves class-conditional feature alignment through an alternating optimization strategy. The feature extractor integrates CNN, LSTM, and Transformer structures, which can effectively capture local temporal features and global dependency information of modulation signals while maintaining a low parameter scale. The dual classifiers guide the model to learn domain-invariant discriminative features by maximizing and minimizing prediction discrepancies, thereby improving cross-domain generalization ability. Based on the RML2016.10a and RML2016.04c datasets, typical cross-domain scenarios with multiple superimposed factors are constructed for experimental validation. The results show that the proposed method achieves stable and excellent recognition performance in complex cross-domain tasks, with a model parameter count of only about 321K, which is significantly lower than that of comparative methods. While ensuring recognition accuracy, DAAMR achieves a good balance between model complexity and computational efficiency, demonstrating strong potential for engineering applications.