Purpose: To understand why some patients respond to immunotherapy but many do not, a clear picture of the tumor microenvironment (TME) of head and neck squamous cell carcinoma (HNSCC) is key. Here we review the current understanding on the immune composition per HNSCC subsite, the importance of the tumor’s etiology and the prognostic power of specific immune cells. Recent Findings: Large cohort data are mostly based on deconvolution of transcriptional databases. Studies focusing on infiltrate localization often entail small cohorts, a mixture of HNSCC subsites, or focus on a single immune marker rather than the interaction between cells within the TME. Summary: Conclusions on the prognostic impact of specific immune cells in HNSCC are hampered by the use of heterogeneous or small cohorts. To move forward, the field should focus on deciphering the immune composition per HNSCC subsite, in powered cohorts and considering the molecular diversity in this disease.