A big reason there's a stigma to sexual fantasies is that a lot of times when they come up, people are encouraged to lie. It's only in the privacy of our own masturbation that we really feel free to let loose.
Once you see a few of these "encouraged to lie" situations, or end up in one yourself, you wind up gunshy about revealing the truth of your sexual experience and desires. I think all the time about an example I encountered when I was pretty young. This was just a show I saw on TV, but it deeply affected me, making me feel dirty about myself for years to follow.
Some women discussed sexual fantasies on a talk show--I can't remember which one. It was set up like a slumber party. The women were chilling out, wearing pajamas, eating chocolate, and chatting "intimately" for their enormous TV audience.
I vividly remember the part where the host asked them about sexual fantasies--with a therapist present, mind you. The women began describing scenes of bathtubs, candles, and massage. "That's good," the therapist cooed. "Why do you say that?" asked one of the women, obviously nervous. I guess she hadn't known her fantasies would be graded. "Well," said the therapist (and I paraphrase), "a lot of women fantasize about rape and things like that. I'm glad none of you are doing that. Maybe it means society is moving forward. It's good to hear these nice, healthy fantasies."
My paraphrase may be unfair--it's been decades, so I'm sure I don't remember the therapist's exact words. However, the paraphrase is true to the message I took away from seeing the show.
And when, lo and behold, I found I had rape fantasies, I knew thanks to the show that these were a sign that I hadn't moved forward with the rest of society. I had the kind of fantasies that would not make her glad, by God. I felt ashamed of myself.
Now, aside from that unfortunate outcome, which I'm sure wasn't just me, I also question whether those women on the show were telling the truth. My experience with fantasies is that when I'm talking about them, I test the waters. I tell a "nice" fantasy, and if the other person doesn't freak out, I step it up a notch, just a little bit.
Believe me, I would never whip out my fantasy of being pierced in seven places on my body as part of a dark sex magic ritual without plenty of assurance that the other person is going to be cool with hearing this. (It's actually easier on a blog than in person, because generally in person the listener is wondering if I'm about to ask them to pull out the stainless steel needles).
My strong suspicion is that the therapist on the show jumped the gun. If the women at the slumber party were reassured and drawn out to deeper levels of expression, I bet they would have come up with much racier stuff. My evidence for this? Just check out Nancy Friday's My Secret Garden, or any other book devoted to collecting actual sexual fantasies, rather than socially acceptable ones.