You are a middle-aged dad, and there is a nubile young female living in your house and caring for your kids. Naturally, there is plenty of proximity, relaxed familiarity, and time together. Over time, it can feel polygamous. All it can take to ignite things sometimes is just a couple of glasses of wine.
I have seen this plenty of times, and do not believe it automatically means that there is a marital problem (and anyway, marriages almost always have some problem or other). When it comes to light, however, it is a big problem indeed.
When Nannies Threaten Marriages