Studies which examined the association between sedentary behavior (SB) and cognitive function have presented equivocal findings. Mentally active/inactive sedentary domains may relate differently to cognitive function. We examined associations between SB and cognitive function, specifically focusing on different domains. Participants were recruited from the Nijmegen Exercise Study 2018 in the Netherlands. SB (h/day) was measured with the Sedentary Behavior Questionnaire. Cognitive function was assessed with a validated computer self-test (COST-A), and a z-score calculated for global cognitive function. Multivariate linear regression assessed associations between tertiles of sedentary time and cognitive function. Cognition tests were available from 2821 participants, complete data from 2237 participants (43% female), with a median age of 61 [IQR 52–67] and a mean sedentary time of 8.3 ± 3.2 h/day. In fully adjusted models, cognitive function was significantly better in participants with the highest total sedentary time (0.07 [95% CI 0.02–0.12], P = 0.01), work-related sedentary time (0.13 [95% CI 0.07–0.19], P < 0.001), and non-occupational computer time (0.07 [95% CI 0.02–0.12], P = 0.01), compared to the least sedentary. Leisure sedentary time and time spent sedentary in the domains TV, reading or creative time showed no association with cognitive function in final models (all P > 0.05). We found a strong, independent positive association between total SB and cognitive function in a heterogenous population. This relation was not consistent across different domains, with especially work- and computer-related SB being positively associated with cognitive function. This highlights the importance of assessing the various sedentary domains in understanding the relation between sedentary time and cognitive function.