Physical activity, alcohol use and smoking might affect cardiovascular disease through modifying autonomic nervous system (ANS) activity. We investigated: 1) whether there are consistent relationships between lifestyle factors and cardiac ANS activity over time, and 2) whether 2-year changes in lifestyle factors relate to 2-year changes in cardiac activity. Baseline (n = 2618) and 2-year follow-up (n = 2010) data of the Netherlands Study of Depression and Anxiety was combined. Baseline data was collected in the Netherlands from 2004–2007. Lifestyle factors were habitual physical activity, frequency of sport activities, alcohol use, and smoking. Indicators of cardiac activity were heart rate (HR), respiratory sinus arrhythmia (RSA) and pre-ejection period (PEP) (100 min of registration). The results showed that high physical activity (− 1.8 beats/min compared to low activity), high frequency of sport activities (‘couple of times/week’: − 2.5 beats/min compared to ‘almost never’) and mild/moderate alcohol use (− 1.2 beats/min compared to non-drinking) were related to low HR. Heavy smoking was related to high HR (> 30 cigarettes/day: + 5.1 beats/min compared to non-smoking). High frequency of sport activities was associated with high RSA (‘couple of times/week’:+1.7 ms compared to ‘almost never’) and moderate smoking with longer PEP (11–20 cigarettes/day: + 2.8 ms compared to non-smoking). Associations were consistent across waves. Furthermore, 2-year change in frequency of sport activities and number of smoked cigarettes/day was accompanied by 2-year change in HR (β = − 0.076 and β = 0.101, respectively) and RSA (β = 0.046 and β = − 0.040, respectively). Our findings support consistent effects of lifestyle on HR and parasympathetic activity in the expected direction. Cardiac autonomic dysregulation may be partly mediating the relationship between lifestyle and subsequent cardiovascular health.