Oncogenic mutations are common in healthy tissues but rarely form tumors. Using the mammary gland as a model, we explored how tissue organization and dynamics affect mutant cell fate. Through lineage tracing and intravital imaging, we found that rounds of local tissue remodeling, driven by the estrous cycle led to the elimination of the majority of mutant clones. However, it simultaneously enabled a minority of mutant clones, that by chance survived, to geometrically expand. This expansion led to cohesive fields of mutant cells spanning large parts of the mammary ducts, predisposing these ducts to tumor formation. Once tumors form, breast cancer response to neoadjuvant chemotherapy varies, even within the same subtype. Again, we identified the estrous cycle as a key factor in this variability. In three mouse models, NAC response was reduced when treatment began during diestrus compared to estrus. Similar patterns were observed in retrospective premenopausal human cohorts, highlighting the importance of treatment timing for improved chemotherapy outcomes.