Backpropagation and stochastic gradient descent method

The backpropagation learning method has opened a way to wide applications of neural network research. It is a type of the stochastic descent method known in the sixties. The present paper reviews the wide applicability of the stochastic gradient descent method to various types of models and loss functions. In particular, we apply it to the pattern recognition problem, obtaining a new learning algorithm based on the information criterion. Dynamical properties of learning curves are then studied based on an old paper by the author where the stochastic descent method was proposed for general multilayer networks. The paper is concluded with a short section offering some historical remarks.