Purpose: To identify which factors are associated with adverse work outcome 5–10 years after diagnosis. Methods: In this cross-sectional study, breast cancer survivors, treated between 2003 and 2008, completed a questionnaire 5–10 years after diagnosis. Adverse work outcome was defined as not having paid employment or working > 20% less compared to prediagnosis. Logistic regression analyses were conducted. Results: Of 906 participants, 326 (36%) had an adverse work outcome. In multivariable analyses, the probability of an adverse work outcome increased with age (OR, 1.03; 95% CI, 1.00–1.07), time since diagnosis (OR, 1.19; 95% CI, 1.03–1.37), and was higher among women who stated that work had become less important (OR, 2.99; 95% CI, 1.94–4.62). Factors associated with a lower probability of an adverse work outcome were having sufficient financial resources (OR, 0.23; 95% CI, 0.08–0.66), higher total work ability (OR, 0.61; 95% CI, 0.54–0.69), feeling supported at work (OR, 0.52; 95% CI, 0.33–0.80), and, prior to diagnosis, having more children to take care of (OR, 0.65; 95% CI, 0.54–0.79), being able to adjust working hours (OR, 0.55; 95% CI, 0.36–0.83) and not desiring to work less hours if that were to be financially feasible (OR, 1.8; 95% CI, 1.0–3.2). Conclusions: Predominantly, work-related factors are associated with adverse work outcomes 5–10 years after diagnosis, whereas clinical factors are not. Our results need validation in prospective cohort studies, after which supportive interventions could be developed. Implications for Cancer Survivors: Work-related factors should be considered in future interventions to prevent adverse work outcome 5–10 years after diagnosis.