Background: Tumor size and metastatic extent may influence tumor response to immunotherapy in non-small cell lung cancer (NSCLC). The aim of this study was to examine the relationship between both baseline sum of longest diameters (bSLD) and number of metastatic organs (NMO) and the tumor response to pembrolizumab. Secondly, we aimed to analyze the association of baseline SLD and NMO with progression-free survival (PFS) and overall survival (OS). Methods: This retrospective study included patients with high PD-L1 expressing tumors (≥50%) and a good performance score (ECOG ≤ 2) that received first-line pembrolizumab monotherapy. Tumor response was calculated as the ‘SLD-change score’ and ‘early treatment discontinuation’ within 3 months on therapy (ETD). The relationship of both bSLD (based on RECIST v1.1) and NMO with tumor response and survival outcome (PFS, OS) was evaluated. Results: No significant differences in SLD-change score could be found using bSLD (OR = 1.010, 95%CI = 0.999–1.021), or using NMO at baseline (OR = 1.608, 95%CI = 0.943–2.743). A bSLD cut-off value of 90 mm was found to be most distinctive for ETD. This cut-off value showed a significant difference for PFS (HR = 2.28, 95%CI = 1.12–4.64, p = 0.023) and OS (HR = 2.99, 95%CI = 1.41–6.34, p = 0.004). NMO also showed a difference for PFS and OS, however, not statistically significant. Conclusions: Tumor size and metastatic extent could not discriminate for tumor response, however, a bSLD of 90 mm could differentiate for PFS and OS.