Objectives This study determined temporal stability of ambulatory measured cardiac autonomic activity for different time periods and investigated potential drivers of changes in this activity. Methods Data was drawn from baseline (n = 2379), 2-year (n = 2245), and 6-year (n = 1876) follow-up from the Netherlands Study of Depression and Anxiety. Cardiac autonomic activity was measured with heart rate (HR), respiratory sinus arrhythmia (RSA) and pre-ejection period (PEP). Autonomic temporal stability was determined across 2, 4, and 6 year intervals. We subsequently examined the association between sociodemographics, lifestyle, mental health, cardiometabolic health, and the use of antidepressant and cardiac medication with change in cardiac autonomic activity. Results Over 2 years, stability was good for HR (ICC = 0.703), excellent for RSA (ICC = 0.792) and moderate for PEP (ICC = 0.576). Stability decreased for a 4- (HR ICC = 0.688, RSA ICC = 0.652 and PEP ICC = 0.387) and 6-year interval (HR ICC = 0.633, RSA ICC = 0.654 and PEP ICC = 0.355). The most important determinants for increase in HR were (increase in) smoking, increase in body mass index (BMI) and (starting) the use of antidepressants. Beta-blocking/antiarrhythmic drug use led to a decrease in HR. Decrease in RSA was associated with age, smoking and (starting) antidepressant use. Decrease in PEP was associated with age and (increase in) BMI. Conclusions Cardiac autonomic measures were rather stable over 2 years, but stability decreased with increasing time span. Determinants contributing to cardiac autonomic deterioration were older age, (increase in) smoking and BMI, and (starting) the use of antidepressants. (Starting) the use of cardiac medication improved autonomic function.