Neural module networks: A review

The capability of deep neural networks to automatically learn informative features from data is their asset. For instance, a convolutional layer learns the filters based on their placement in the architecture, i.e., the low-level features in the early layers and more abstract features in the higher level. The question is whether we can learn at the sub-task level and automate the process. In other words, can a neural architecture learn to decompose a complex task into sub-tasks (i.e., elemental tasks), solve each sub-task, and aggregate the results? This way, we gain full transparency and explainability of how a complex task has been solved. This is the goal of neural module networks (NMN). Each module represents a sub-task that conveys a symbolic meaning. Each module learns the assigned sub-task based on its placement in the modules’ layout, internal architecture, or both. The NMN re-shapes itself for each sample by choosing the sample-specific modules (i.e., sub-tasks) and placing them into an appropriate layout. This review provides a comprehensive overview of neural module networks and their applications and assumes little prior knowledge. We showcased different applications of NMNs and compared their various implementations. To better compare the performance of each application, we also chose a few non-modular approaches for the completeness of the comparisons. We hope this review and the added benefit of NMN’s explainability attract more researchers to solve the current challenges.