Designing mixture of deep experts

Mixture of Experts (MoE) is a classical architecture for ensembles where each\nmember is specialised in a given part of the input space or its expertise area.\nWorking in this manner, we aim to specialise the experts on smaller problems,\nsolving the original problem through some type of divide and conquer approach.\nThe goal of our research is to initially reproduce the work done by Collobert et\nal[1] , 2002 followed by extending this work by using neural networks as experts\non different datasets. Specialised representations will be learned over different\naspects of the problem, and the results of the different members will be merged\naccording to their specific expertise. This expertise can then be learned itself by a\ngiven network acting as a gating function.\nMOE architecture composed on N expert networks. These experts are combined\nvia a gating network, which partition the input space accordingly. It is based on\ndivide and conquer strategy supervised by a gating network. Using a specialised\ncost function the experts specialise in their sub-space. Using the discriminative\npower of experts is much better than simply clustering. The gating network needs\nto needs to learn how to assign examples to different specialists.\nSuch models show promise for building larger networks that are still cheap to\ncompute at test time, and more parallelizable at training time. We were able to\nreproduce the work by the author and implemented a multi-class gater to classify\nimages.\nWe know that Neural Networks perform the best with lots of data. However,\nsome of our experiments require us to divide the dataset and train multiple Neural\nNetworks. We observe that in data deprived condition our MoE are almost on\npar and compete with ensembles trained on complete data.\nKeywords : Machine Learning, Multi Layer Perceptrons, Mixture of Experts, Support\nVector Machines, Divide and Conquer, Stochastic Gradient Descent, Optimization.