Older adults frequently report pain; cross-sectional studies have shown that pain is associated with worse cognitive function. However, longitudinal studies are lacking. We prospectively studied 441 participants without dementia, including 285 with pain, aged 65 years and older, enrolled in the Central Control of Mobility in Aging study, a prospective cohort study. We analyzed the longitudinal association between pain (measured with the Medical Outcomes Study pain severity scale) and major cognitive impairment (measured with the Repeatable Battery for the Assessment of Neuropsychological Status and the Trail Making Test Delta) using Cox regression analysis adjusted for age, gender, ethnicity, and education. Over a mean follow-up of 2.75 years (standard deviation = 1.94), there was no difference in the risk of developing cognitive impairment between participants with pain and participants without pain. However, among those with pain, risk for developing major memory impairment was higher among those with high levels of pain than those with low levels of pain (adjusted hazard ratio = 3.47, 95% confidence interval = 1.42–8.46). The association with pain and incident impairments in attention or executive function was not significant. We did not find that pain is associated with incident cognitive impairment in general, but among older adults with pain, a high level of pain is associated with increased risk of developing incident memory impairment. Perspective: Our study results suggest that high levels of pain may contribute to incident memory impairment. Further research is needed to determine whether a high level of chronic pain is a modifiable risk factor for cognitive impairment in older adults.