Late Night Fantasies at Work...

I'm married with kids, but lately something's been gnawing at me. At work, I've started fantasizing about a colleague while working the night shift... it just feels so wrong yet oddly relieving. We've flirted in passing but never anything more. Last week when we were both closing up late, my imagination got away from me and I found myself practically panting over him behind closed doors. He doesn't know what he does to me. It's not like me at all, being so... drawn in by these kinds of thoughts around someone else. Is it just a phase? Should I talk to him or ignore this temptation before it leads somewhere regrettable? It feels so taboo and I'm torn between my marriage and the allure...