常微分方程
颂歌
人工神经网络
计算机科学
等价(形式语言)
前馈
前馈神经网络
过程(计算)
残余物
数学优化
微分方程
算法
数学
人工智能
应用数学
控制工程
操作系统
离散数学
工程类
数学分析
作者
Dong Pang,Xinyi Le,Xinchun Guan,Jun Wang
出处
期刊:IEEE transactions on neural networks and learning systems
[Institute of Electrical and Electronics Engineers]
日期:2022-01-01
卷期号:: 1-10
标识
DOI:10.1109/tnnls.2022.3213308
摘要
Since the last decade, deep neural networks have shown remarkable capability in learning representations. The recently proposed neural ordinary differential equations (NODEs) can be viewed as the continuous-time equivalence of residual neural networks. It has been shown that NODEs have a tremendous advantage over the conventional counterparts in terms of spatial complexity for modeling continuous-time processes. However, existing NODEs methods entail their final time to be specified in advance, precluding the models from choosing a desirable final time and limiting their expressive capabilities. In this article, we propose learnable final-time (LFT) NODEs to overcome this limitation. LFT rebuilds the NODEs learning process as a final-time-free optimal control problem and employs the calculus of variations to derive the learning algorithm of NODEs. In contrast to existing NODEs methods, the new approach empowers the NODEs models to choose their suitable final time, thus being more flexible in adjusting the model depth for given tasks. Additionally, we analyze the gradient estimation errors caused by numerical ordinary differential equations (ODEs) solvers and employ checkpoint-based methods to obtain accurate gradients. We demonstrate the effectiveness of the proposed method with experimental results on continuous normalizing flows (CNFs) and feedforward models.
科研通智能强力驱动
Strongly Powered by AbleSci AI