Cancer cells can leverage several cell-intrinsic and -extrinsic mechanisms to escape immune system recognition. The inherent complexity of the tumor microenvironment, with its multicellular and dynamic nature, poses great challenges for the extraction of biomarkers of immune response and immunotherapy efficacy. Here, we use RNA-sequencing (RNA-seq) data combined with different sources of prior knowledge to derive system-based signatures of the tumor microenvironment, quantifying immune-cell composition and intra- and intercellular communications. We applied multi-task learning to these signatures to predict different hallmarks of immune responses and derive cancer-type-specific models based on interpretable systems biomarkers. By applying our models to independent RNA-seq data from cancer patients treated with PD-1/PD-L1 inhibitors, we demonstrated that our method to Estimate Systems Immune Response (EaSIeR) accurately predicts therapeutic outcome. We anticipate that EaSIeR will be a valuable tool to provide a holistic description of immune responses in complex and dynamic systems such as tumors using available RNA-seq data.