White matter hyperintensities (WMH) constitute the visible spectrum of cerebral small vessel disease (SVD) markers and are associated with cognitive decline, although they do not fully account for memory decline observed in individuals with SVD. We hypothesize that WMH might exert their effect on memory decline indirectly by affecting remote brain structures such as the hippocampus. We investigated the temporal interactions between WMH, hippocampal atrophy and memory decline in older adults with SVD. Five hundred and three participants of the RUNDMC study underwent neuroimaging and cognitive assessments up to 3 times over 8.7 years. We assessed WMH volumes semi-automatically and calculated hippocampal volumes (HV) using FreeSurfer. We used linear mixed effects models and causal mediation analyses to assess both interaction and mediation effects of hippocampal atrophy in the associations between WMH and memory decline, separately for working memory (WM) and episodic memory (EM). Linear mixed effect models revealed that the interaction between WMH and hippocampal volumes explained memory decline (WM: β =.067; 95%CI[.024–0.111]; p <.01; EM: β =.061; 95%CI[.025–.098]; p <.01), with better model fit when the WMH*HV interaction term was added to the model, for both WM (likelihood ratio test, χ2 = 9.3, p <.01) and for EM (likelihood ratio test, χ2 = 10.7, p <.01). Mediation models showed that both baseline WMH volume (β = −.170; p =.001) and hippocampal atrophy (β = 0.126; p =.009) were independently related to EM decline, but the effect of baseline WMH on EM decline was not mediated by hippocampal atrophy (p value indirect effect: 0.572). Memory decline in elderly with SVD was best explained by the interaction of WMH and hippocampal volumes. The relationship between WMH and memory was not causally mediated by hippocampal atrophy, suggesting that memory decline during aging is a heterogeneous condition in which different pathologies contribute to the memory decline observed in elderly with SVD.