When you abandon it, or just take a break or a vacation.
Everything goes downhill quickly - cardio, endurance, speed, strength. Measurably downhill in 10 days. Unfortunately, things do not ratchet up in a positive direction unless you keep pushing it. In fact, there is no ratchet. There is only uphill or downhill.
That's why Mrs. BD and I try to stick to a fitness regime even on vacation. Too much time and money invested in it to let it slip away. Time off is like blowing your savings.