On an instant-classic episode of Family Guy, cartoon paterfamilias Peter Griffin gets so perilously in touch with his feminine side that he tries to nurse his baby son, Stewie. Asleep when his proud papa draws him to his bared nipple, Stewie suckles instinctively but soon awakens, groggily fishing a little strand of stray chest hair out of his mouth and then recoiling in a spasm of sputtering horror.

I know how Stewie feels. No, my dad never tried to nurse me, but when I’m among certain men, I get the feeling that maybe they want to. Why else would they so triumphantly thrust their nipples at me through fitted shirts?

But then I wonder whether this is just my issue. After all, what’s wrong with nipples? Entire fashion and porn empires have been built on reminding us that women have them. Every guy has them too (Mark Wahlberg, famously, even has three). So if nipples are sexy on women, why aren’t they sexy on men? And how can something inherent to the male body seem so off-puttingly, weirdly feminine—and distractingly kinky when seen in public or, worse, at the office?

In the past, male nipples were rarely displayed, because every man wore an undershirt. Outside of the boudoir, the locker room, or the beach, it never even occurred to anyone that men had tits. Then fashion took a turn for the worse. “We basically stopped selling undershirts in the nineties,” says Debbie Nelson, a sales consultant at Savile Row, an upscale haberdasher in St. Louis. Nelson has tracked men’s fashion in the heartland for the past 25 years. “It’s just not a staple anymore.”

Meanwhile, menswear sales have been skewing over the past decade toward “more fitted styles, not only in button-downs and dress shirts but in sweaters and T-shirts and polo shirts,” says Melissa Payner-Gregor, president and CEO of the online clothing retailer Bluefly.com. “Guys are just showing off their bodies more.” For this we can blame gym culture, as well as Abercrombie & Fitch, whose endless catalog spreads of bare-breasted corn-fed dudes seem to have emboldened the general public.

Still, the question remains: Do women find male nipples sexy? “It’s usually not where I’m looking,” admits Robin Adams, one of the founders of Sweet Action, the “porn for girls” magazine that celebrates un-buff, average cute guys. “But I suppose if a guy has on a nice tight T-shirt with visible nips, I might have a hard time not staring. As far as it being a turn-on, that totally depends on the guy.” Her colleague Hope Gangloff adds, “Guys wearing shirts tight enough to display their nips—we might assume they’re the type of guys who wouldn’t notice ours.

And that, when you get down to it, is the real problem with extroverted male nipples. They set up a gratuitous competition—one that no guy, no matter how pec’d out, can ever win. Gender equality aside, women really are better at a lot of stuff, including looking totally hot in tight shirts. And breast-feeding. Which perhaps suggests a good rule about nipple display: If you can actually use it, flaunt it. Otherwise, maybe look into buying some Hanes Beefy-Ts.