While there are minimal sex differences in overall intelligence, males, on average, have larger total brain volume and corresponding regional brain volumes compared to females, measures that are consistently related to intelligence. Limited research has examined which other brain characteristics may differentially contribute to intelligence in females to facilitate equal performance on intelligence measures. Recent reports of sex differences in the neural characteristics of the brain further highlight the need to differentiate how the structural neural characteristics relate to intellectual ability in males and females. The current study utilized a graph network approach in conjunction with structural equation modeling to examine potential sex differences in the relationship between white matter efficiency, fronto-parietal gray matter volume, and general cognitive ability (GCA). Participants were healthy adults (n = 244) who completed a battery of cognitive testing and underwent structural neuroimaging. Results indicated that in males, a latent factor of fronto-parietal gray matter was significantly related to GCA when controlling for total gray matter volume. In females, white matter efficiency and total gray matter volume were significantly related to GCA, with no specificity of the fronto-parietal gray matter factor over and above total gray matter volume. This work highlights that different neural characteristics across males and females may contribute to performance on intelligence measures. Hum Brain Mapp 37:4006–4016, 2016.