Aim: To assess the associations of age and diagnosis with visual ratings of medial temporal lobe atrophy (MTA), parietal atrophy (PA), global cortical atrophy (GCA), and white matter hyperintensities (WMH) and to investigate their clinical value in a large memory clinic cohort. Methods: We included 2,934 patients (age 67 ± 9 years; 1,391 [47%] female; MMSE 24 ± 5) from the Amsterdam Dementia Cohort (1,347 dementia due to Alzheimer's disease [AD]; 681 mild cognitive impairment [MCI]; 906 controls with subjective cognitive decline). We analyzed the effect of age, APOE e4 and diagnosis on visual ratings using linear regression analyses. Subsequently, we compared diagnostic and predictive value in three age-groups (< 65 years, 65-75 years, and > 75 years). Results: Linear regression analyses showed main effects of age and diagnosis and an interaction age*diagnosis for MTA, PA, and GCA. For MTA the interaction effect indicated steeper age effects in MCI and AD than in controls. PA and GCA increased with age in MCI and controls, while AD patients have a high score, regardless of age. For WMH we found a main effect of age, but not of diagnosis. For MTA, GCA and PA, diagnostic value was best in patients < 65 years (optimal cut-off: ≥1). PA and GCA only discriminated in patients < 65 years and MTA in patients < 75 years. WMH did not discriminate at all. Taking into account APOE did not affect the identified optimal cut-offs. When we used these scales to predict progression in MCI using Cox proportional hazard models, only MTA (cut-off≥2) had any predictive value, restricted to patients > 75 years. Conclusion: Visual ratings of atrophy and WMH were differently affected by age and diagnosis, requiring an age-specific approach in clinical practice. Their diagnostic value seems strongest in younger patients.