It's 11 a.m. on Sunday, a time that during your single days was reserved for sex or the gym. But for your newly grown-up, coupled-up crowd it means . . . brunch. Now that most of your friends are over 35 and some have children, this kids-'n'-coffee routine is beginning to feel pleasantly familiar. Until he shows up—the guy who's never been married. He's late, fresh from the gym, and accompanied by a woman who's about the same age and build as the aspiring-actress waitress.

You used to envy this man. Sitting there with his hand on a 23-year-old's thigh while he sips his latte, he makes your banana-pancake domestic life feel lame. But lately that guy's beginning to seem—to you, your friends, and your wife—well, kind of creepy. His brazen rejection of the life stage that most of his peers have gotten to is starting to make it look like there's something wrong with him.

Joe (who asked that only his first name be used), a 39-year-old union organizer in New York who's never been married, has been getting disapproving looks from his friends ever since he turned 30. "There is nothing like a group of married people—especially with kids—when you come into their circle with a younger, thin woman," he says. "It's a terrible reaction."

"These guys get labeled playboy, loser, commitment-phobe," says Carl Weisman, author of So Why Have You Never Been Married? According to U.S. Census Bureau statistics, in 1980 only 6 percent of men between 40 and 44 had never been married; in 2008 it was 16 percent. But even though there are more of them around, men with long-term single status still have a hard time explaining their situation to potential dates, who see a guy entering middle age without ever having been married as damaged goods. In fact, a man whose marriage failed spectacularly tends to arouse less suspicion than a straight, still-single 41-year-old. "If he's over 40, you would hope that he's divorced," says Janis Spindel, a high-end matchmaker in New York who gets calls from hundreds of single women asking for setups. Evidence that even unmarried men in their mid-thirties are suspect is in her fee structure: The up-front charge for guys under 35 is $25,000; for those 35-plus it's $50,000.

If you ask a guy in his late thirties or early forties why he isn't married, he'll have his answer—you could call it his defense—ready. For some, the rationale is a self-fulfilling prophecy. Joe tends to date women younger than he, who are less likely to want to settle down than those his own age. "I would still like to have kids," he says. "But if I date someone who's 40, it's going to be chaos, a lot of pressure if we want kids—because we have to start that immediately, and even then you're not guaranteed."