反例
点(几何)
正多边形
失真(音乐)
人工神经网络
计算机科学
数学
数学优化
算法
人工智能
组合数学
几何学
计算机网络
放大器
带宽(计算)
作者
Kiwook Bae,Han-Youl Ryu,Hayong Shin
出处
期刊:Cornell University - arXiv
日期:2019-11-01
被引量:1
摘要
The adaptive optimizer for training neural networks has continually evolved to overcome the limitations of the previously proposed adaptive methods. Recent studies have found the rare counterexamples that Adam cannot converge to the optimal point. Those counterexamples reveal the distortion of Adam due to a small second momentum from a small gradient. Unlike previous studies, we show Adam cannot keep closer to the optimal point for not only the counterexamples but also a general convex region when the effective learning rate exceeds the certain bound. Subsequently, we propose an algorithm that overcomes Adam's limitation and ensures that it can reach and stay at the optimal point region.
科研通智能强力驱动
Strongly Powered by AbleSci AI