It's just
Dean apologised twice for kicking Cas out of the bunker. Once, they had just killed Ephraim; Cas proved himself useful and skilled as a hunter. The second time, he had just gotten his powers back. I'm not saying Dean did it on purpose, but Cas would think that Dean only did it because he was useful again, not because he really ment it, definitely not because he was forced to send him away.
After, he starts to lose his power, he gets weaker and Sam hurts his arm on a hunt together. Cas' solution to being a liability to the Winchesters is to isolate himself in a motel room, probably to die there too. Again, Sam didn't think of it that way, I don't even think he knew Cas was dying.
Years later, the angels are almost all dead and it gets harder and harder to do anything with his grace. He tells the Winchesters that something's wrong with it, they ignore him. Or he thinks so, because after that Dean is absolutely furious with him and he knows a lot has to do with Mary and Jack, Rowena and Belphagor, but he can't help but wonder if him being more human is the proverbial last straw.
Dean forgives him once more, which is great, really. He still had to deal with grief, anger, despair, coldness, loneliness, hunger all alone, again. He still was homeless, again. He knows it's his own fault, he had it coming, he's the one who left when they needed him the most but it doesn't matter. He needed them more anyway and they made it pretty clear that they wouldn't want broken things like him to stick around.
When he dies it's a sacrifice. He is happy to be useful for the last time.
