Mean teachers are better role models: Weight-averaged consistency\n targets improve semi-supervised deep learning results

The recently proposed Temporal Ensembling has achieved state-of-the-art\nresults in several semi-supervised learning benchmarks. It maintains an\nexponential moving average of label predictions on each training example, and\npenalizes predictions that are inconsistent with this target. However, because\nthe targets change only once per epoch, Temporal Ensembling becomes unwieldy\nwhen learning large datasets. To overcome this problem, we propose Mean\nTeacher, a method that averages model weights instead of label predictions. As\nan additional benefit, Mean Teacher improves test accuracy and enables training\nwith fewer labels than Temporal Ensembling. Without changing the network\narchitecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250\nlabels, outperforming Temporal Ensembling trained with 1000 labels. We also\nshow that a good network architecture is crucial to performance. Combining Mean\nTeacher and Residual Networks, we improve the state of the art on CIFAR-10 with\n4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels\nfrom 35.24% to 9.11%.\n