Background Fatigue is a disabling problem in patients with visual impairment, but its etiology is still poorly understood. Our objective was to identify the determinants of fatigue in adults with visual impairment compared to adults with normal sight. Methods Cross-sectional data on fatigue and sociodemographic, psychological and health-related factors was obtained with validated questionnaires. Structural equational modeling using hypothesized relationships and explorative analyses were used to identify (in)direct pathways contributing to fatigue in 247 adults with visual impairment. The model was then tested in a reference group of 151 adults with normal sight. Results The final model explained 64% of fatigue variance in participants with visual impairment and revealed the following factors to be directly associated with fatigue: depressive symptoms (β = 0.723, p<0.001), perceived health (β = -0.158, p = 0.004), accommodative coping (β = 0.116, p = 0.030) and somatic comorbidity (β = 0.311, p = 0.001). Self-efficacy demonstrated a beneficial indirect effect on fatigue (β = -0.228, p<0.001) mediated by depression, accommodative coping and perceived health. Sleep disorder had an indirect effect on fatigue (β = 0.656, p<0.001) mediated by depression and perceived health. After removal of sleep disorder, the model explained 58% of the fatigue variance in normally sighted adults but pathways involving accommodative coping and somatic comorbidity were not confirmed. Conclusions These findings suggest that depression and perceived health are important mediating factors that contribute to fatigue in persons with visual impairment and normal sight. In contrast, somatic comorbidity, sleep disorders and accommodative coping seem to have a specific contribution to vision-related fatigue. These factors should be addressed in interventions to assist individuals with visual impairment in dealing with fatigue.