Childhood adversity plays an important role for development of major depressive disorder (MDD). There are differences in subcortical brain structures between patients with MDD and healthy controls, but the specific impact of childhood adversity on such structures in MDD remains unclear. Thus, aim of the present study was to investigate whether childhood adversity is associated with subcortical volumes and how it interacts with a diagnosis of MDD and sex. Within the ENIGMA-MDD network, nine university partner sites, which assessed childhood adversity and magnetic resonance imaging in patients with MDD and controls, took part in the current joint mega-analysis. In this largest effort world-wide to identify subcortical brain structure differences related to childhood adversity, 3036 participants were analyzed for subcortical brain volumes using FreeSurfer. A significant interaction was evident between childhood adversity, MDD diagnosis, sex, and region. Increased exposure to childhood adversity was associated with smaller caudate volumes in females independent of MDD. All subcategories of childhood adversity were negatively associated with caudate volumes in females - in particular emotional neglect and physical neglect (independently from age, ICV, imaging site and MDD diagnosis). There was no interaction effect between childhood adversity and MDD diagnosis on subcortical brain volumes. Childhood adversity is one of the contributors to brain structural abnormalities. It is associated with subcortical brain abnormalities that are relevant to psychiatric disorders such as depression.