Background: Men with the hereditary peroxisomal disorder X-linked adrenoleukodystrophy (ALD) are at risk of developing inflammatory demyelinating lesions in the brain. In the absence of inflammatory (post-contrast enhancing) lesions on MRI cognitive function is considered spared, but some form of cognitive dysfunction may nevertheless be present. The aim of this cross-sectional study was to characterize cognitive functioning of ALD men with no or minimal MRI abnormalities, which will define cognitive functioning in this category of patients. Methods: A neuropsychological battery covering a broad range of cognitive domains, including language, verbal and non-verbal memory, visuoconstruction, executive functioning, and psychomotor speed, was used. Means and proportions of borderline and impaired T scores ≤36 were compared to the standardized norm group and a qualitative case-by-case analysis was performed for participants with T scores ≤36 within ≥2 domains. Patients with MRI abnormalities that were extensive (Loes score > 3) or showed enhancement post-contrast were excluded. Results: Thirty-three men participated (median age 44 years, range 19-71). Mean performance on verbal fluency was poorer in patients (45.70 ± 8.85 patients vs. 50 ± 10 standardized norm group, p = 0.009), as was the percentage of borderline and impaired scores on visuoconstruction (Beery VMI: 19% patients vs. 8% standardized norm group, p = 0.02; RCFT copy: 81% patients vs. 2% standardized norm group, p < 0.0005) and mental reaction time during a complex decision task (18% patients vs. 8% standardized norm group, p = 0.055). Moreover, 9/33 (27.3%) patients had T scores ≤36 within ≥2 domains. Conclusions: Given the heterogeneous pattern of mostly borderline scores cognitive functioning seems not impaired in the vast majority of adult ALD males with no or minimal MRI abnormalities. However, borderline to impaired cognitive dysfunction was present in 27.3%, with the majority being borderline scores. Longitudinal studies will have to determine if this reflects early cerebral disease under the detection limit of MRI.