conflicted about my kink

i have this thing where i really like the idea of a stranger or someone id never see again being in control and treating me however they want. i dont know why, maybe it just feels freeing to not be responsible for how things go down. my partner gets into some stuff but hes not super comfortable with all of it, especially if we both know who this person is. i met this guy online once though, he was really sweet and understanding, and when i told him about this fantasy he said he got it totally. we decided to try it out for real one night while my partner was asleep at home. it went so well that the next day he messaged me wanting to do more. now its been a few weeks and hes still texting me every once in a while asking if i want to get together again, but i dont know how to tell him no without sounding like an asshole. hes cool about everything being totally consensual and all, which is great, but it just feels weird not having my partner there. im starting to wonder if this is really such a good idea.