Multi-Scale Spatio-Temporal Fusion With Adaptive Brain Topology Learning for fMRI Based Neural Decoding

Neural decoding aims to extract information from neurons' activities to reveal how the brain functions. Due to the inherent spatial and temporal characteristics of brain signals, spatio-temporal computing has become a hot topic for neural decoding. However, the extant spatio-temporal decoding methods usually use static brain topology, ignoring the dynamic patterns of the interaction between brain regions. Further, they do not identify the hierarchical organization of brain topology, leading to only superficial insight into brain spatio-temporal interactions. Therefore, here we propose a novel framework, the Multi-Scale Spatio-Temporal framework with Adaptive Brain Topology Learning (MSST-ABTL), for neural decoding. It includes two new capabilities to enhance spatio-temporal decoding: i) ABTL module, which learns dynamic brain topology while updating specific patterns of brain regions, ii) MSST module, which captures the association of spatial pattern and temporal evolution, and further enhances the interpretability of the learned dynamic topology from multi-scale perspective. We evaluated the framework on the public Human Connectome Project (HCP) dataset (resting-state and task-related fMRI data). The extensive experiments show that the proposed MSST-ABTL outperforms state-of-the-art methods on four evaluation metrics, and also can renew the neuroscientific discoveries in the brain's hierarchical patterns.