残余物
一般化
人工神经网络
边距(机器学习)
上下界
正规化(语言学)
数学
计算机科学
残差神经网络
人工智能
算法
机器学习
数学分析
作者
Fenghua He,Tongliang Liu,Dacheng Tao
出处
期刊:IEEE transactions on neural networks and learning systems
[Institute of Electrical and Electronics Engineers]
日期:2020-12-01
卷期号:31 (12): 5349-5362
被引量:107
标识
DOI:10.1109/tnnls.2020.2966319
摘要
Residual connections significantly boost the performance of deep neural networks. However, few theoretical results address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This article studies the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space. We first present an upper bound of the covering number of networks with residual connections. This bound shares a similar structure with that of neural networks without residual connections. This result suggests that moving a weight matrix or nonlinear activation from the bone to a vine would not increase the hypothesis space. Afterward, an O(1 / √N) margin-based multiclass generalization bound is obtained for ResNet, as an exemplary case of any deep neural network with residual connections. Generalization guarantees for similar state-of-the-art neural network architectures, such as DenseNet and ResNeXt, are straightforward. According to the obtained generalization bound, we should introduce regularization terms to control the magnitude of the norms of weight matrices not to increase too much, in practice, to ensure a good generalization ability, which justifies the technique of weight decay.
科研通智能强力驱动
Strongly Powered by AbleSci AI