Introduction: In multiple sclerosis (MS), brain atrophy measurement on magnetic resonance imaging (MRI) reflects overall tissue loss, especially demyelination and axonal loss. We studied which factor contributes most to the development of brain atrophy: extent and severity of lesions or damage of whole brain tissue (WBT). Methods: Eighty-six patients with MS [32 primary progressive (PP), 32 secondary progressive (SP)] and 22 relapsing-remitting (RR) were studied. MRI included T1- and T2-weighted imaging to obtain hypointense T1 lesion volume (TILV) and two brain volume measurements: 1) the parenchymal fraction (PF; whole brain parenchymal volume/intracranial volume) as a marker of overall brain volume, and 2) the ventricular fraction (VF; ventricular volume/intracranial volume) as a marker of central atrophy. From magnetization transfer ratio (MTR) histograms, the relative peak height (rHp) was derived as an index of damage of WBT (a lower peak height reflects damage of WBT). Results: Multiple linear regression analysis revealed that damage of WBT explains most of the variance of PF (standardized coefficient β=0.59, p<0.001 for WBT and β=-0.19, p<0.05 for TILV). These findings are independent of disease phase; even in RR patients, damage of WBT plays a dominant role in explaining the variance in overall brain volume. By contrast, the variance in VF is explained by both TILV and damage of WBT (standardized coefficient β=0.43, p<0.001 for TILV and β=-0.38, p<0.001 for WBT). Conclusion: This study shows that overall brain volume (PF) is best explained by damage of WBT, supporting the significance of nonfocal pathology in MS in producing tissue loss. Central atrophy (VF) is determined by both lesion volume and damage of WBT. Our results underline the importance of nonfocal pathology even in the early (RR) phase of the disease.