Who do you tell?

?

Today I was casually talking to one of my co-workers when I slipped up about the acupuncture appointment. It naturally worked into conversation. We were talking about drinks for another co-worker, to which I'd already RSVPd yes, and I was telling him that it's actually a "no". He asked why and, before I could catch myself, I said "I have my acupuncture appointment that day".
"Acupuncture?" he inquired "why the heck do you need acupuncture?" And before I knew it, I was lying "You know I get terrible migraines... maybe it'll help."
When he walked away, I felt like such a loser. Did I really have to lie about my infertility? Why couldn't I just be honest. But a part of me felt like it wasn't really information I need to share with him. While I am close with this particular co-worker, I don't want to have to talk to him about my girl parts, working or not. He's the same co-worker who jokes that I'm pregnant all the time.When he does, I say "Trust me, I'm not". I wish people knew more about infertility and it didn't carry so much of a stigma. Maybe the stigma is in my head. Either way, I'm not quite ready to talk about it with every random person in my life.
So, how did you distinguish who to tell and who not to?