Estrogen receptor positive breast neoplasias represent over 70% of diagnosed breast cancers. Depending on the stage at which the tumor is detected, HER2 status and genomic risk, endocrine therapy is combined with either radio, chemo and/or targeted therapy. A growing amount of evidence supports the notion that components of the tumor microenvironment play specific roles in response to treatment and that strategies targeting these key interactions with tumor cells could pave the way to a new generation of therapies. In this review, we analyze the evidence suggesting different components of the tumor microenvironment play a role in hormone receptor positive breast cancer progression. In particular we focus on the immune system, carcinoma associated fibroblasts and the extracellular matrix. Further insight into the cross talk between these constituents of the microenvironment and the tumor cells may lead to therapies that eliminate disseminated metastatic cells early on, and thus reduce distant disease relapse which is the leading cause of death for patients who are diagnosed with this illness.