Background Increasing numbers of adolescents seek help for gender-identity questions. Consequently, requests for medical treatments, such as puberty suppression, are growing. However, studies investigating the neurobiological substrate of gender incongruence (when birth-assigned sex and gender identity do not align) are scarce, and knowledge about the effects of puberty suppression on the developing brain of transgender youth is limited. Methods Here we cross-sectionally investigated sex and gender differences in regional fractional anisotropy (FA) as measured by diffusion MR imaging, and the impact of puberty on alterations in the white-matter organization of 35 treatment-naive prepubertal children and 41 adolescents with gender incongruence, receiving puberty suppression. The transgender groups were compared with 79 age-matched, treatment-naive cisgender (when sex and gender align) peers. Results We found that transgender adolescents had lower FA in the bilateral inferior fronto-occipital fasciculus (IFOF), forceps major and corpus callosum than cisgender peers. In addition, average FA values of the right IFOF correlated negatively with adolescents' cumulative dosage of puberty suppressants received. Of note, prepubertal children also showed significant FA group differences in, again, the right IFOF and left cortico-spinal tract, but with the reverse pattern (transgender > cisgender) than was seen in adolescents. Conclusions Importantly, our results of lower FA (indexing less longitudinal organization, fiber coherence, and myelination) in the IFOF of gender-incongruent adolescents replicate prior findings in transgender adults, suggesting a salient neural correlate of gender incongruence. Findings highlight the complexity with which (pubertal) sex hormones impact white-matter development and add important insight into the neurobiological substrate associated with gender incongruence.