Background: Imbalance in the autonomic nervous system due to a disrupted circadian rhythm may be a cause of shift work-related cardiovascular diseases. Objective: We aimed to determine the association between shift work and cardiac autonomic activity in blue-collar workers. Methods: The study included 665 blue-collar workers aged 18–68 years in different occupations from two Danish cohort studies. Time and frequency domain parameters of heart rate variability (HRV) were measured during sleep using the Actiheart monitor, and used as markers of cardiac autonomic function. Multiple linear regression analyses were used to investigate differences in HRV between day and shift workers. Results: Shift workers had no significantly different HRV parameters than day workers, except for a lower VLF (B: 0.21; 95% CI: −0.36–0.05). The lower VLF was only present among non-night shift workers (p < 0.05) and not among night shift workers (p > 0.05). Results differed significantly by gender (p for interaction < 0.10): among men, shift work was negatively associated with RMSSD (B: −7.83; 95% CI: −14.28–1.38), SDNN (B: −7.0; 95% CI: −12.27–1.78), VLF (B: −0.27; 95% CI: −0.46–0.09) and Total Power (B: −0.61; 95% CI: −1.20–0.03), while among women, shift work was only associated with the LF/HF ratio (B: −0.29; 95% CI: −0.54–0.03). Conclusion: Shift work was particularly associated with lower HRV during sleep among men. This indicates that shift work causes imbalance in the autonomic nervous system among men, which might increase their risk of cardiovascular diseases.