Background: Evidence for the diet-depression link is growing but longitudinal studies on the reverse association are scarce. We investigated associations of (1) current depressive symptoms, (2) short-term changes in and (3) long-term history of depressive symptoms with three a priori diet quality indices. Methods: Data were from participants (≥ 55 years) of the Longitudinal Aging Study Amsterdam (LASA). The Mediterranean Diet Score (MDS), Alternative Healthy Eating Index (AHEI-2010) and Dietary Approaches to Stop Hypertension diet (DASH) were derived in 2014/2015. Depressive symptoms (Center for Epidemiologic Studies Depression scale; CES-D) were assessed in 2014/2015 and at five regular 3-yearly cycles from 2001–2003 to 2015/2016. Associations between three depression determinants and the diet indices were analysed by multivariable linear regression models. Results: Cross-sectionally (n = 1312), current depressive symptoms (CES-D ≥ 16) were associated with lower MDS (adjusted B = −1.21, 95%CI = −2.41, −0.023) and AHEI (B = −2.72, 95%CI = −5.24, −0.20) scores in men only. Chronic/recurrent depressive symptoms (CES-D ≥ 16 in both 2011–2013 and 2015/2016) were associated with lower MDS scores (n = 1233; B = −2.22, 95%CI = −3.40, −1.04) and a trend for lower AHEI scores (B = −2.37, 95%CI = −4.92, 0.18), compared to no depressive symptoms (twice CES-D < 16). History of depressive symptoms (ever CES-D ≥ 16 from 2001–2003 to 2011–2013; n = 687) was associated with lower MDS (B = −1.87, 95%CI = −3.47, −0.27) and AHEI (B = −4.33, 95%CI = −7.54, −1.13) scores in men only. No associations were found with the DASH score. Limitations: Single dietary data collection impeded investigation of prospective depression-diet associations. Conclusions: Our study in middle-aged and older adults suggests that current and past depressive symptoms are associated with poorer diet quality, particularly in men.