MMF-NNs: Multi-modal Multi-granularity Fusion Neural Networks for brain networks and its application to epilepsy identification

Structural and functional brain networks are generated from two scan sequences of magnetic resonance imaging data, which can provide different perspectives for describing pathological changes caused by brain diseases. Recent studies found that fusing these two types of brain networks improves performance in brain disease identification. However, traditional fusion models combine these brain networks at a single granularity, ignoring the natural multi-granularity structure of brain networks that can be divided into the edge, node, and graph levels. To this end, this paper proposes a Multi-modal Multi-granularity Fusion Neural Networks (MMF-NNs) framework for brain networks, which integrates the features of the multi-modal brain network from global (i.e., graph-level) and local (i.e., edge-level and node-level) granularities to take full advantage of the topological information. Specifically, we design an interactive feature learning module at the local granularity to learn feature maps of structural and functional brain networks at the edge-level and the node-level, respectively. In that way, these two types of brain networks are fused during the feature learning process. At the global granularity, a multi-modal decomposition bilinear pooling module is designed to learn the graph-level joint representation of these brain networks. Experiments on real epilepsy datasets demonstrate that MMF-NNs are superior to several state-of-the-art methods in epilepsy identification.