Background Although several classifications of perihilar cholangiocarcinoma (PHC) include vascular involvement, its prognostic value has not been investigated. Our aim was to assess the prognostic value of unilateral and main/bilateral involvement of the portal vein (PV) and hepatic artery (HA) on imaging in patients with PHC. Methods All patients with PHC between 2002 and 2014 were included regardless of stage or management. Vascular involvement was defined as apparent tumor contact of at least 180° to the PV or HA on imaging. Kaplan–Meier method with log-rank test was used to compare overall survival (OS) between groups. Cox regression was used for multivariable analysis. Results In total, 674 patients were included with a median OS of 12.2 (95% CI 10.6–13.7) months. Patients with unilateral PV involvement had a median OS of 13.3 (11.0–15.7) months, compared with 14.7 (11.7–17.6) in patients without PV involvement (p = 0.12). Patients with main/bilateral PV involvement had an inferior median OS of 8.0 (5.4–10.7, p < 0.001) months. Median OS for patients with unilateral HA involvement was 10.6 (9.3–12.0) months compared with 16.9 (13.2–20.5) in patients without HA involvement (p < 0.001). Patients with main/bilateral HA involvement had an inferior median OS of 6.9 (3.3–10.5, p < 0.001). Independent poor prognostic factors included unilateral and main/bilateral HA involvement, but not PV involvement. Conclusion Both unilateral and main HA involvement are independent poor prognostic factors for OS in patients presenting with PHC, whereas PV involvement is not.