The role of disturbed chromatin remodeling in the pathogenesis of intellectual disability (ID) is well established and illustrated by de novo mutations found in a plethora of genes encoding for proteins of the epigenetic regulatory machinery. We describe mutations in the “SET nuclear proto-oncogene” (SET), encoding a component of the “inhibitor of histone acetyltransferases” (INHAT) complex, involved in transcriptional silencing. Using whole exome sequencing, four patients were identified with de novo mutations in the SET gene. Additionally, an affected mother and child were detected who carried a frameshift variant in SET. Four patients were found in literature. The de novo mutations in patients affected all four known SET mRNA transcripts. LoF mutations in SET are exceedingly rare in the normal population and, if present, affect only one transcript. The pivotal role of SET in neurogenesis is evident from in vitro and animal models. SET interacts with numerous proteins involved in histone modification, including proteins encoded by known autosomal dominant ID genes, that is, EP300, CREBBP, SETBP1, KMT2A, RAC1, and CTCF. Our study identifies SET as a new component of epigenetic regulatory modules underlying human cognitive disorders, and as a first member of the Nucleosome Assembly Protein (NAP) family implicated in ID.