Keeping My Secret Life From Him

I met my husband when I was in my mid-twenties and transitioning. At the time, I told him everything about myself, all of my history, but there was one thing I couldn't bring myself to say. It's this secret that's haunted me for years now. He's a good man, really caring and supportive. We've been together through surgeries, counseling sessions, hormone therapy, every step. When we married, it felt like everything had fallen into place. But here’s the thing: there was always this one part of my past I kept hidden. Back in high school, I used to go to these parties where a lot of older guys would be around. It's not that they all did anything illegal or harmful, but it felt... wrong somehow. There were some really intense nights that I never told anyone about, even before him. And now that we're married? The idea of telling him sends shivers down my spine. I didn't want to risk our relationship, so I just said nothing. He knows I was a late bloomer, but he doesn’t know the extent of what happened or how it shaped me back then. Part of me wonders if he ever suspected something and let it go, or if he's completely clueless. Nowadays, we have a healthy sex life that's consensual, fun, and loving, nothing illegal or abusive there at all. But this secret is eating away at my conscience. I feel so conflicted; every time I think about sharing with him, I freeze up. What do you think? Am I the asshole for keeping something from him for so long?