Objective: We hypothesized that in participants with a history of hypertension, lower late-life blood pressure (BP) will be associated with more brain pathology. Methods: Participants are 4,057 older men and women without dementia with midlife (mean age 50 ± 6 years) and late-life (mean age 76 ± 5 years) vascular screening, cognitive function, and brain structures onMRI ascertained as part of the Age, Gene/Environment Susceptibility (AGES)- Reykjavik Study. Results: The association of late-life BP to brain measures depended on midlife hypertension history. Higher late-life systolic and diastolic BP (DBP) was associated with an increased risk of white matter lesions and cerebral microbleeds, and this was most pronounced in participants without a history of midlife hypertension. In contrast, in participants with a history of midlife hypertension, lower late-life DBP was associated with smaller total brain and gray matter volumes. This finding was reflected back in cognitive performance; in participants with midlife hypertension, lower DBP was associated with lower memory scores. Conclusion: In this large population-based cohort, late-life BP differentially affects brain pathology and cognitive performance, depending on the history of midlife hypertension. Our study suggests history of hypertension is critical to understand how late-life BP affects brain structure and function.