Purpose: Adherence to the Mediterranean diet has been associated with fewer depressive symptoms, however, it is unknown whether this is attributed to some or to all components. We examined the association between the individual food groups of the Mediterranean Diet Score (MDS), in isolation and in combination, with depression and anxiety (symptom severity and diagnosis). Methods: Data from 1634 adults were available from the Netherlands Study of Depression and Anxiety. Eleven energy-adjusted food groups were created from a 238-item food frequency questionnaire. In regression analysis, these were associated in isolation and combination with (1) depressive and anxiety disorders (established with the Composite International Diagnostic Interview) (current disorder n = 414), and (2) depression and anxiety severity [measured with the Inventory of Depressive Symptomatology (IDS), the Beck Anxiety Inventory (BAI) and the Fear Questionnaire (FEAR)]. Results: Overall, the MDS score shows the strongest relationships with depression/anxiety [Diagnosis: odds ratio (OR) 0.77 per SD, 95% confidence interval (95% CI) 0.66–0.90, IDS: standardised betas (β) − 0.13, 95% CI − 0.18, − 0.08] and anxiety (BAI: β − 0.11, 95% CI − 0.16, − 0.06, FEAR: β − 0.08, 95% CI − 0.13, − 0.03). Greater consumption of non-refined grains and vegetables was associated with lower depression and anxiety severity, whilst being a non-drinker was associated with higher symptom severity. Higher fruit and vegetable intake was associated with lower fear severity. Non-refined grain consumption was associated with lower odds and being a non-drinker with greater odds of current depression/anxiety disorders compared to healthy controls, these associations persisted after adjustment for other food groups (OR 0.82 per SD, 95% CI 0.71–0.96, OR 1.26 per SD 95% CI 1.08–1.46). Conclusion: We can conclude that non-refined grains, vegetables and alcohol intake appeared to be the driving variables for the associated the total MDS score and depression/anxiety. However, the combined effect of the whole diet remains important for mental health. It should be explored whether an increase consumption of non-refined grains and vegetables may help to prevent or reduce depression and anxiety.