Simultaneous information maximization and minimization

In this paper, we propose a method to unify information maximization and minimization methods so far developed independently to control internal representations. Information maximization methods have been applied to the interpretation of internal representations. On the other hand information minimization methods have been used to improve generalization performance. Thus, if it is possible to maximize and minimize information, interpretation and generalization performance can simultaneously be improved. To maximize and simultaneously minimize information, we propose networks with two hidden layers. In one layer, information is forced to be maximum, while in another layer, information is decreased as much as possible. The method with two layers was applied to the simple XOR problem. Experimental results confirmed that information can be maximized and simultaneously minimized. In addition, simpler internal representations could be obtained.