it's important to me that sam's college friends have various theories about his childhood but are reluctant to gossip about what they figure must have been some pretty horrific abuse behind his back, so there's just kind of this black cloud around sam's childhood that everyone is quietly judging and making assumptions about without talking to him, and everything they say to each other just builds on it until it's this huge gnarly mass he can't escape from.
he doesn't even mention dean unless he's really seriously drunk, and everyone has different ideas about him, ranging from him having died tragically, to him and sam having had an incestuous relationship that sam ran away from. sam gets really defensive when people try to poke at the subject of dean, and for a lot of his friends it's the only time they've seen him properly outwardly angry, which just fuels the fire.
when it gets around that sam just up and left with this mysterious (not-dead) brother to look for their dad without a word after jess died, everyone freaks, and for once they're all saying the things usually deemed too ugly to say out loud right into sam's voicemail, in an effort to beg sam back home.
for at least a week sam is fielding constant calls about everything his friends have been thinking about him and his family his entire college career, but had been too polite to say aloud. it's a slap in the face, a bucket of cold water in the morning, a sudden reminder that he was never normal, and despite doing his very best to shed his previous life, it clung to him like a second skin, and everyone who spent more than a day around him could smell it on him.
it gets worse when they return to palo alto to take down the shapeshifter. sam might have been able to play-act at normal without dean, but he knows it's hopeless when they're together. he's scared they'll see it on him like a brand. he's not even sure what it is. this sick jumble of feelings in his chest when he looks at his brother? the preternatural way they know each other, that casts light on how irregular they are in all other interaction? the trained violence that lives in their skin?
previously, the only people ever in their orbit long enough to see him in any real capacity were victims and other hunters. and he knows with certainty that without the immediate threat of death or the liberal relationship with "normal" that makes one turn to hunting, he'll be pinned like a beetle, on display.