Background: Neuroimaging studies have demonstrated gray matter (GM) volume abnormalities in substance users. While the majority of substance users are polysubstance users, very little is known about the relation between GM volume abnormalities and polysubstance use. Methods: In this study we assessed the relation between GM volume, and the use of alcohol, tobacco, cocaine and cannabis as well as the total number of substances used, in a sample of 169 males: 15 non-substance users, 89 moderate drinkers, 27 moderate drinkers who also smoke tobacco, 13 moderate drinkers who also smoke tobacco and use cocaine, 10 heavy drinkers who smoke tobacco and use cocaine and 15 heavy drinkers who smoke tobacco, cannabis and use cocaine. Results: Regression analyses showed that there was a negative relation between the number of substances used and volume of the dorsal medial prefrontal cortex (mPFC) and the ventral mPFC. Without controlling for the use of other substances, the volume of the dorsal mPFC was negatively associated with the use of alcohol, tobacco, and cocaine. After controlling for the use of other substances, a negative relation was found between tobacco and cocaine and volume of the thalami and ventrolateral PFC, respectively. Conclusion: These findings indicate that mPFC alterations may not be substance-specific, but rather related to the number of substances used, whereas, thalamic and ventrolateral PFC pathology is specifically associated with tobacco and cocaine use, respectively. These findings are important, as the differential alterations in GM volume may underlie different cognitive deficits associated with substance use disorders.