Caught Off Guard at My Friend's House Party

I met Jack last month when we both attended a friend’s house party. I’ve been married for five years now but my husband has been away on deployment, and I haven’t really gone out much lately. So, meeting someone new felt like it might be refreshing. Jack is in his early thirties, similar age to me, and we clicked immediately. He’s handsome, has a good sense of humor, and just seemed easy-going and fun. We spent the evening chatting and drinking some wine. As the night wore on, he started getting more flirtatious with me. We kissed for the first time when I was walking him to his car at the end of the party. He lingered behind my door when I went inside to go to bed that night, pressing a little harder against me than is comfortable. At first, it felt exciting, but then I pulled back and told him that I should probably just get some sleep. The next day, Jack texted asking if we could hang out again soon. We’ve gone on a couple of casual dates since, where things got more physical each time, hand-holding in the park, kissing by the lake, and one night at his place ended with him undoing my bra button and us making out heavily. What’s been bothering me is that this feels like a pattern. He hasn’t asked for anything serious yet, but he seems to want more every time we get together. I haven’t given up on finding someone to date seriously, but there are days when I feel torn between exploring what we have and hoping my husband will return soon to pick things back up. Yesterday at his place, Jack wanted me to touch him a little more than he had before. It was consensual, I let him pull off my shirt, ran my hands over his chest, but after that, I pulled back again. He got this look in his eye like he wanted something deeper, and it scared me. I haven’t talked about any of this with Jack yet because honestly, I’m not sure how to approach the topic. Am I the asshole for playing along so far? Is there a way to tell him that things need to slow down or stop without hurting his feelings? What would you do if you were me?