Background: The impact of cardiovascular risk burden on cognitive trajectories and brain structure changes remains unclear. Objectives: This study aimed to examine whether cardiovascular risk burden assessed by the Framingham General Cardiovascular Risk Score (FGCRS) is associated with cognitive decline and structural brain differences. Methods: Within the Rush Memory and Aging Project, 1,588 dementia-free participants (mean age: 79.5 years) were followed for up to 21 years. FGCRS was assessed at baseline and categorized into tertiles (lowest, middle, and highest). Episodic memory, semantic memory, working memory, visuospatial ability, and perceptual speed were assessed annually with a battery of 19 tests, from which composite scores were derived. A subsample (n = 378) of participants underwent magnetic resonance imaging. Structural total and regional brain volumes were estimated. Data were analyzed using linear mixed-effects models and linear regression models. Results: In all participants, FGCRS ranged from 4 to 28 (mean score: 15.6 ± 3.7). Compared with the lowest tertile of FGCRS, the highest tertile was associated with faster decline in global cognition (β = −0.019; 95% confidence interval [CI]: −0.035 to −0.003), episodic memory (β = −0.023; 95% CI: −0.041 to −0.004), working memory (β = −0.021; 95% CI: −0.035 to −0.007), and perceptual speed (β = −0.027; 95% CI: −0.042 to −0.011) over the follow-up. In magnetic resonance imaging data analyses, higher FGCRS was related to smaller volumes of the hippocampus (β = −0.021; 95% CI: −0.042 to −0.000), gray matter (β = −1.569; 95% CI: −2.757 to −0.382), and total brain (β = −1.588; 95% CI: −2.832 to −0.344), and greater volume of white matter hyperintensities (β = 0.035; 95% CI: 0.001 to 0.069). Conclusions: Higher cardiovascular risk burden may predict decline in episodic memory, working memory, and perceptual speed and is associated with neurodegeneration and vascular lesions in the brain.