Purpose: Initial dose of chemotherapy is planned based on body surface area, which does not take body composition into account. We studied the association between fat mass (kg and relative to total body weight) as well as lean mass (kg and relative to total body weight) and toxicity-induced modifications of treatment in breast cancer patients receiving chemotherapy. Methods: In an observational study among 172 breast cancer patients (stage I–IIIB) in the Netherlands, we assessed body composition using dual-energy X-ray scans. Information on toxicity-induced modifications of treatment, defined as dose reductions, cycle delays, regimen switches, or premature termination of chemotherapy, was abstracted from medical records. Adjusted hazard ratios and 95% confidence intervals (95% CI) were calculated to assess associations between body composition and the risk of toxicity-induced modifications of treatment. Results: In total, 95 out of 172 (55%) patients experienced toxicity-induced modifications of treatment. Higher absolute and relative fat mass were associated with higher risk of these modifications (HR 1.14 per 5 kg; 95% CI 1.04–1.25 and HR 1.21 per 5%; 95% CI 1.05–1.38, respectively). A higher relative lean mass was associated with a lower risk of modifications (HR 0.83 per 5%; 95% CI 0.72–0.96). There was no association between absolute lean mass and risk of toxicity-induced modifications of treatment. Conclusions: A higher absolute and a higher relative fat mass was associated with an increased risk of toxicity-induced modifications of treatment. Absolute lean mass was not associated with risk of these treatment modifications, while higher relative lean mass associated with lower risk of modifications. These data suggest that total fat mass importantly determines the risk of toxicities during chemotherapy in breast cancer patients.