Self-Adaptive Weight Based on Dual-Attention for Differentiable Neural Architecture Search

Differentiable architecture search is a popular gradient-based method for neural architecture search, and has achieved great success in automating design of neural network architectures. However, it still has some limitations such as performance collapse, which seriously affects network architecture performance. To solve this issue, we propose an algorithm called self-adaptive weight based on dual-attention for differentiable neural architecture search (SWD-NAS) in this article. SWD-NAS utilizes a dual-attention mechanism to measure architectural weights. Specifically, an upper-attention module is used to adaptively select channels based on their weights before inputting into the search space. A lower-attention (LA) module is utilized to calculate architectural weights. In addition, we propose an architectural weight normalization to alleviate the unfair competition among connection edges. Finally, we evaluate the architectures searched on CIFAR-10 and CIFAR-100, achieving test errors of 2.51% and 16.13%, respectively. Furthermore, we transfer the architecture searched on CIFAR-10 to ImageNet, achieving top-one and top-five errors of 24.5% and 7.6%, respectively. This demonstrates the superior performance of the proposed algorithm compared to many gradient-based algorithms.