Mutual information maximization for improving and interpreting multi-layered neural networks

The present paper aims to propose a new type of information-theoretic method to maximize mutul information between neurons. The importance of mutual information has been well known in neural networks, but the actual implementation of mutual information maximization is a hard problem and mutual information has not necessarily been used in neural networks. We can say that the application of mutual information is very limited. To overcome this shortcoming of mutual information maximization, we present here the very simplified version of mutual information maximization by supposing that mutual information is already maximized before learning. The method was applied to the wholesale data set and the inference of default credit card holders. The experimental results show that mutual information between neurons could be increased and generalization performance could be improved. Then, the important features can be obtained by the present method, even if the training data set was small. On the other hand, by the logistic regression analysis, the important features could be extracted only with the large training data set.