Abstract Background and Purpose Mild Cognitive Impairment (MCI) is associated with a greater risk of dementia for older adults. However, systematic reviews have shown that some physical exercise (PE) seems to improve MCI symptoms and signs. Those reviews and meta-analysis could not explain what possible moderator influenced their results. This meta-analysis aims to identify the effect of PE over older people’s cognition with MCI and explore sources of heterogeneity. Methods Databases were searched from inception January 2020 for randomized clinical trials that evaluated the effects of PE over cognition of older persons with MCI. Random effect meta-analyses were performed for each cognitive outcome. Subgroup analyses and meta-regressions models explored the potential sources of heterogeneity. Results A total of 2077 participants (mean age = 71.8 years) from 27 studies were included. PE improves global cognitive function (SMD = 0.348 [95 % CI 0.166 to 0.529]; p = 0.0001), executive function (SMD = 0.213 [95 % CI 0.026 to 0.400]; p = 0.026) and delayed recall (SMD = 0.180 [95 % CI 0.002 to 0.358]; p = 0.047). A trend towards beneficial effects of PE on verbal fluency (SMD = 0.270 [95 %, CI -0.021 to 0.561]; p = 0.069) and attention (SMD = 0.170 [CI -0.016 to 0.357]; p = 0.073) were also observed. Subgroup analyses showed a relationship between modality and intensity of physical exercise and changes observed in global cognitive function, executive function, delayed recall, verbal fluency and working memory. Discussion and Conclusion PE can ameliorate cognitive deficts of older adults with MCI. The most pronounced effects appear to arise from other types of exercise that included mind-body exercises and moderate intensity.