Frailty is a common syndrome in older individuals that is associated with poor cognitive outcome. The underlying brain correlates of frailty are unclear. The aim of this study was to investigate the association between frailty and MRI features of cerebral small vessel disease in a group of non-demented older individuals. We included 170 participants who were classified as frail (n = 30), pre-frail (n = 85) or non-frail (n = 55). The association of frailty and white matter hyperintensity volume and shape features, lacunar infarcts and cerebral perfusion was investigated by regression analyses adjusted for age and sex. Frail and pre-frail participants were older, more often female and showed higher white matter hyperintensity volume (0.69 [95%-CI 0.08 to 1.31], p = 0.03 respectively 0.43 [95%-CI: 0.04 to 0.82], p = 0.03) compared to non-frail participants. Frail participants showed a non-significant trend, and pre-frail participants showed a more complex shape of white matter hyperintensities (concavity index: 0.04 [95%-CI: 0.03 to 0.08], p = 0.03; fractal dimensions: 0.07 [95%-CI: 0.00 to 0.15], p = 0.05) compared to non-frail participants. No between group differences were found in gray matter perfusion or in the presence of lacunar infarcts. In conclusion, increased white matter hyperintensity volume and a more complex white matter hyperintensity shape may be structural brain correlates of the frailty phenotype.