Mechanism for Enhancement of Functionality in Deep Echo State Network by Optimizing Leaking Rate

Deep echo state network (Deep-ESN) model consists of multiple reservoir layers, that can respond to layer-specific different time-scales. This dynamical characteristic leads to enhance performance of ESN. However, neither the design guidelines for the hyperparameters of the network and individual neurons nor the mechanism to produce the diverse dynamical response have been clarified. In this study, we proposed an approach to generate the dynamical responses with different time-scales in each layer by adjusting the leaking rate of neurons. Through the evaluations time-series prediction task for different leaking rates, multiscale entropy analysis for each reservoir layer, and cross-correlation between adjacent layers, we found that when the leaking rate is set to low, the layer-specific dynamics with different time-scales are generated, as well as a mechanism whereby the signal propagates to subsequent layers with a delay, i.e., queue characteristic. These characteristics produce a high memory capacity. Consequently, diverse responses with delay leads to an enhancement of Deep-ESN functionality.