Comprehensive analysis of tumor infiltrating myeloid cells in the tumor microenvironment of penile squamous cell carcinoma (PSCC) is lacking. In this retrospective study, for the first time, PSCC resection specimens (N = 103) were annotated into the following compartments: intratumoral tumor (IT Tumor), intratumoral stroma (IT Stroma), peritumoral tumor (PT Tumor) and peritumoral stroma (PT Stroma) compartments. We then quantified CD14+, CD68+ and CD163+ myeloid cells within these compartments using an image analysis software and assessed their association with various clinical parameters, including high-risk human papillomavirus (hrHPV) status. In the total cohort, hrHPV status, grade of differentiation, age and tumor size were associated with myeloid cell densities. hrHPV+ tumors had higher infiltration rates of CD14+, CD68+ and CD163+ myeloid cells in the IT tumor compartment (p < 0.001, for all) compared to hrHPV- tumors. Furthermore, when examining the association between compartment-specific infiltration and differentiation grade, increased myeloid cell densities in the IT tumor compartment were associated with a more advanced histological grade (p < 0.001, for all). This association remained significant when the hrHPV- cohort (N = 60) was analyzed (CD14+ p = 0.001; CD68+ p < 0.001; CD163+ p = 0.004). Subgroup analysis in the hrHPV+ group (N = 43) showed that high infiltration rates of CD68+ and CD163+ cells in the PT tumor compartment were associated with lymph node (LN) metastasis (p = 0.031 and p = 0.026, respectively). Regarding the association between myeloid cell densities and disease-specific survival, the risk of death was found to decrease slightly as the number of myeloid cells in the IT tumor compartment increased (CD14+ p = 0.04; CD68+ p = 0.05; CD163+ p = 0.02). However, after adjusting for hrHPV, no independent association between myeloid densities and disease-specific survival were found. Altogether, these findings demonstrate the importance of assessing myeloid cell densities within the spatial context of the tumor. Further studies are needed to unravel the specific phenotype of myeloid cells residing in the different compartments, their effect on clinical parameters and the impact of hrHPV on the recruitment of myeloid cell populations in PSCC.