初始化
计算机科学
块(置换群论)
卷积神经网络
推论
算法
非线性系统
人工智能
模式识别(心理学)
数学
几何学
程序设计语言
量子力学
物理
作者
Linzhuo Yang,Lan Zhang
出处
期刊:
[Institute of Electrical and Electronics Engineers]
日期:2022-08-01
卷期号:: 19-28
被引量:1
标识
DOI:10.1109/bigcom57025.2022.00011
摘要
We propose a new structure called Expansion-Squeeze-Block by leveraging over-parameterization to train given compact neural networks. The structure expands the width of convolutional layers and adds shortcut connections for better performance without adding any nonlinearity. The expanded networks can be contracted back to the original format algebraically at inference time without loss of information. In addition, we introduce a new initialization method utilizing the weights of the pretrained original networks, which can further improve the accuracy. We evaluate our methods on CIFAR-100 and Tiny-ImageNet datasets to show their effectiveness. As evidenced by our experiments, Expansion-Squeeze-Block outperforms the baselines in most cases with average 0.81% improvements and 1.15% combined with our initialization method on CIFAR-100 and Tiny-ImageNet datasets. We also demonstrate the effectiveness of the partial expansion strategy by choosing the layers of the highest importance in VGG11-BN0.5x and ResNet20. We can save on average 58.82% #MACs and 38.82% #Params than expanding all layers.
科研通智能强力驱动
Strongly Powered by AbleSci AI