god i just love heart. it's such a good episode. sam doesn't want madison to die because madison is an allegory for himself and sam does not want to die. he doesn't want to be a monster. he sees himself in madison and he desperately wants her to be good because if she is good then so too, maybe, is he. and killing madison is admitting that there's no hope for him and that he can't be saved.
what's particularly thrilling about this episode though is the way this allegory interacts with dean, and particularly the developing relationship between sam and dean.
namely, sam refuses to let dean be the one to kill madison. he takes up the mantle and expressly disallows dean from being there when the bullet is fired. throughout the episode, even, dean is sent on the wild goose chase while sam stays close to madison. and oftentimes, these chases are initiated by sam, who urges dean to follow the trail. it comes off, in the end, as if sam is trying to actively prevent dean from interfering with what he needs to do. which is, ultimately, to kill madison.
when sam begs dean not to be the one who kills madison, he is ultimately talking about himself. the message of the episode is that he doesn't want dean to kill him. this is a striking difference from playthings, where sam begged dean to kill him. sam spends much of this span of time between the two episodes grappling with the depth of dean's love and his fears of the inevitable (his own monstrosity). and the turning point of these episodes appears to be the direct midpoint between 2x11 and 2x17, born under a bad sign—it's where dean insists that he will save sam no matter what happens. and from then on, sam's fears turn from the inevitability of his monstrosity to the desperate hope that dean will, in fact, save him. his dependence on dean for stability and hope is growing, and he comes to trust in dean as the only person on earth who can save him from the promise of the monster.
madison is therefore a conduit through which this discussion about their fate is had: if dean is successful in killing madison, then it becomes a confirmation of sam's fears. if dean kills madison, he is killing sam. he's going back on his promise to sam. it's a betrayal of the highest degree, and sam must prevent that at any cost. and so he disarms dean and sends him away from madison, never lets him get too close. and in the end, when he realizes his efforts were futile, he kills madison himself. metaphorically, he kills himself before dean can do it for him, which is a subversion of fate in itself. he may not be able to prevent the monster, but he can destroy it with his own power, and destiny will be thwarted because dean's hands will be clean.
and as such, this is a microcosm, a foreshadowing, of swan song. where the metaphor becomes real and the monster inside of sam is indeed killed by sam's own hands, leaving dean's bloodless. more than anything, 2x17 heart insists, sam will not let fate run its course. if he can't prevent his own fate, then he will prevent dean's. so he kills madison; so he throws himself into the pit. it's the ultimate act in service of the brother and therefore the ultimate act against destiny. and well i just think that's neat.