Purpose: The purpose of the study was to assess self-reported taste and smell perception after chemotherapy in breast cancer patients compared with women without cancer, and to assess whether taste and smell perception is associated with quality of life after the end of chemotherapy. Methods: We included 135 newly diagnosed breast cancer patients who completed chemotherapy and 114 women without cancer. Questionnaires on taste, smell, and quality of life were completed shortly after and 6 months after chemotherapy (patients) or at two moments with 6 months’ time window in between (comparisons). Results: Self-reported taste and smell perception were significantly lower in patients shortly after chemotherapy compared to the comparison group. Most patients recovered 6 months after chemotherapy, although patients who were still receiving trastuzumab then reported a lower taste and smell perception compared to patients who were not. A lower self-reported taste and smell were statistically significantly associated with a worse quality of life, social, emotional, and role functioning shortly after chemotherapy. Six months after chemotherapy, taste and smell were statistically significantly associated with quality of life, social and role functioning, but only in patients receiving trastuzumab. Conclusions: Most taste and smell alterations recovered within 6 months after the end of chemotherapy for breast cancer, but not for patients receiving trastuzumab. These results highlight the importance of monitoring taste and smell alterations during and after treatment with chemotherapy and trastuzumab, as they may impact quality of life.