Background Cognitive impairment and depression often co-occur in older adults, but it is not clear whether depression is a risk factor for cognitive decline, a psychological reaction to cognitive decline, or whether changes in depressive symptoms correlate with changes in cognitive performance over time. The co-morbid manifestation of depression and cognitive impairment may reflect either a causal effect or a common cause, depending on the specific symptoms experienced and the cognitive functions affected. Method The study sample comprised 1506 community-dwelling older adults aged ≥65 years from the Longitudinal Aging Study Amsterdam (LASA). We conducted cross-domain latent growth curve analyses to examine longitudinal associations between late-life depression dimensions (i.e. depressed affect, positive affect, and somatic symptoms) and specific domains of cognitive functioning (i.e. processing speed, inductive reasoning, immediate recall, and delayed recall). Results Poorer delayed recall performance at baseline predicted a steeper increase in depressed affect over time. Steeper decline in processing speed correlated with a steeper increase in somatic symptoms of depression over time. Conclusions Our findings suggest a prospective association between memory function and depressed affect, whereby older adults may experience an increase in depressed affect in reaction to poor memory function. Somatic symptoms of depression increased concurrently with declining processing speed, which may reflect common neurodegenerative processes. Our findings do not support the hypothesis that depression symptoms may be a risk factor for cognitive decline in the general population. These findings have potential implications for the treatment of late-life depression and for the prognosis of cognitive outcomes.