Abstract Prognostic factors, such as the Human Epidermal growth factor Receptor 2 (HER2) and Estrogen Receptor (ER) influence distant recurrence‐free survival (RFS) in breast cancer. This study aims to evaluate the interaction between HER2 and ER status with RFS, and if that interaction influences where the metastasis is located. To do this, we used a study population of all women diagnosed with non‐metastatic, invasive breast cancer in Stockholm from 2007 to 2020. Flexible parametric survival models were used to estimate time‐varying survival and hazard ratios (HR) for RFS. Cumulative incidence was used to quantify rates of metastasis in key locations. We found significant interactions between ER and HER2 for RFS ( p = 0.037), which was time varying ( p = 0.017). For ER+ patients, adjusted short‐term survival at 2.5 years after diagnosis was identical for HER2+ compared to HER2− patients (HR 1.02, CI; 0.76–1.39), but was dramatically better for HER2+ patients after 5 years (HR at 7.5 years 0.29, CI; 0.14–0.58). In contrast, among ER− patients, HER2+ patients experienced constant risk compared to HER2− from diagnosis until the end of the study (HR ~0.50). Finally, we observed that HER2+ patients have a higher rate of first metastasis to the brain than HER2− patients ( p < 0.001). Our study demonstrates that the interaction between ER and HER2 status has a time‐varying impact on RFS and plays a role in determining the location of metastasis. Thus, the utilization of complex models that combine ER and HER2 status can enhance the understanding of patient RFS and the likelihood of metastasis in specific locations.