Cognitive training (CT) is an increasingly popular, non-pharmacological intervention for improving cognitive functioning in neurodegenerative diseases and healthy aging. Although meta-analyses support the efficacy of CT in improving cognitive functioning, the neural mechanisms underlying the effects of CT are still unclear. We performed a systematic review of literature in the PubMed, Embase and PsycINFO databases on controlled CT trials (N > 20) in aging and neurodegenerative diseases with pre- and post-training functional MRI outcomes up to November 23rd 2018 (PROSPERO registration number CRD42019103662). Twenty articles were eligible for our systematic review. We distinguished between multi-domain and single-domain CT. CT induced both increases and decreases in task-related functional activation, possibly indicative of an inverted U-shaped curve association between regional brain activity and task performance. Functional connectivity within ‘cognitive’ brain networks was consistently reported to increase after CT while a minority of studies additionally reported increased segregation of frontoparietal and default mode brain networks. Although we acknowledge the large heterogeneity in type of CT, imaging methodology, in-scanner task paradigm and analysis methods between studies, we propose a working model of the effects of CT on brain activity and connectivity in the context of current knowledge on compensatory mechanisms that are associated with aging and neurodegenerative diseases.