Cortisol is an important stress hormone affected by a variety of biological and environmental factors, such as the circadian rhythm, exercise and psychological stress. Cortisol is mostly measured using blood or saliva samples. A number of genetic variants have been found to contribute to cortisol levels with these methods. While the effects of several specific single genetic variants is known, the joint genome-wide contribution to cortisol levels is unclear. Our aim was to estimate the amount of cortisol variance explained by common single nucleotide polymorphisms, i.e. the SNP heritability, using a variety of cortisol measures, cohorts and analysis approaches. We analyzed morning plasma (n = 5705) and saliva levels (n = 1717), as well as diurnal saliva levels (n = 1541), in the Rotterdam Study using genomic restricted maximum likelihood estimation. Additionally, linkage disequilibrium score regression was fitted on the results of genome-wide association studies (GWAS) performed by the CORNET consortium on morning plasma cortisol (n = 12,597) and saliva cortisol (n = 7703). No significant SNP heritability was detected for any cortisol measure, sample or analysis approach. Point estimates ranged from 0% to 9%. Morning plasma cortisol in the CORNET cohorts, the sample with the most power, had a 6% [95%CI: 0–13%] SNP heritability. The results consistently suggest a low SNP heritability of these acute and short-term measures of cortisol. The low SNP heritability may reflect the substantial environmental and, in particular, situational component of these cortisol measures. Future GWAS will require very large sample sizes. Alternatively, more long-term cortisol measures such as hair cortisol samples are needed to discover further genetic pathways regulating cortisol concentrations.