bear in mind, Charlotte's father's estate is substantial, and not entailed away; spinsterhood for her would be quite comfortable
This is factually incorrect.
Sir William Lucas was a tradesman who made a good living and "a tolerable fortune" (not a good fortune, a tolerable fortune) and then sold everything to buy an estate and live a life of leisure when he got made a knight because he made a speech the king liked (chapter 5). Being made a knight gave him all sorts of ideas about what was suitable because (although an amiable man) he is also a huge snob. He would not have sold his business and bought an estate if he hadn't been made a knight, because he would have been much better off to continue in trade. The Lucases are not the Bingleys, they didn't quit trade because they had enough wealth to last for generations.
The Lucas family income from their estate + the money left from Sir William's business dealings is less than the Bennet family income. Exact figures are never stated, but it's implied in a lot of places throughout the story. For example, in chapter 9, Mrs. Bennet tells Mr. Bingley that Charlotte has to help with some of the cooking/food preparation in ways that the Bennet girls don't, because the Bennets have better (and probably more) servants. The Bennet girls don't have to work like the Lucas girls do, because the Bennets can afford not to. Remember what a snob Sir William is, and that he is willing to sacrifice income so that they will be "gentry" and not work. If there was any way he could afford enough servants so the girls didn't have to help in the kitchen, he would do it as a point of pride. (Note: the Bennet girls would still be doing a lot of work within the house; sewing clothes, mending things, making a lot of the nicer decorative things in the Bennet household, etc., etc. But they do NOT do any of the heavier work of cleaning, and they do NO cooking whatsoever. But the Collins girls help out in the kitchen sometimes....)
The other thing to remember, when comparing Elizabeth's situation to Charlotte's, is the Gardiners. Mr. Gardiner is in trade, which is a slight social step down from the Bennets' status as landed gentry. BUT. Mr. Gardiner makes a shit-ton of money. He has enough money to take lengthy, expensive vacations and invite Elizabeth along; and he has enough money that Mr. Bennet thinks it's believable that he paid at least five times the Bennet annual income to get Wickham to marry Lydia. Mr. Gardiner is LOADED. And we know he really loves his nieces, especially Jane and Elizabeth. He's got enough money that supporting them would be no sacrifice, and he loves them enough to do it without begrudging it. No such wealthy and compassionate relation is ever hinted at for the Lucases. For all that Mrs. Bennet wails about their fate after Mr. Bennet dies, the worst possible outcome for Elizabeth is much better than Charlotte's most likely outcome if she didn't marry Mr. Collins.
Chapter 22 gives the Lucas family reaction to Charlotte's engagement: "The whole family in short were properly overjoyed on the occasion. The younger girls formed hopes of coming out a year or two sooner than they might otherwise have done; and the boys were relieved from their apprehension of Charlotte’s dying an old maid. Charlotte herself was tolerably composed. She had gained her point, and had time to consider of it. Her reflections were in general satisfactory. Mr. Collins, to be sure, was neither sensible nor agreeable: his society was irksome, and his attachment to her must be imaginary. But still he would be her husband. Without thinking highly either of men or of matrimony, marriage had always been her object: it was the only honourable provision for well-educated young women of small fortune, and, however uncertain of giving happiness, must be their pleasantest preservative from want."
Small fortune, that's how the text describes her. Which, in this case, means "still in the gentry class but without the money to back it up." She's not a revolutionary, she doesn't want to overthrow society, she doesn't want to be rich, she just wants an "honorable"/respectable place in society, and she wants to be "preserved from want." Getting married is the only way she can have a decent life, and even marrying a man she doesn't like is better than the life she will have if she doesn't marry. And look at her family reaction! They all know who Mr. Collins is, how fatuous and self-centered and boring and condescending! They know as well as Charlotte does what kind of a man he is! Not one of them gives a shit what her life with him is going to be like. Elizabeth is the only person in the entire book who is shown to care what Charlotte's feelings are, and what her life with him will be like.
And sure, Sir William's estate is not entailed! That doesn't mean he's going to divide it among his children. It barely supports one family in the appropriate style. If it's divided, it's not going to support any of them. English tradition is primogeniture (and Sir William is a very traditional man): if there are sons, the ENTIRE estate goes to the eldest son. If there's enough money, the younger sons get an education for a middle-class profession so they can support themselves and the girls get dowries. The younger siblings have to fend for themselves (with hopefully enough to launch them); only the oldest son gets the inheritance, which means that the inheritance gets preserved through the generations. But the younger siblings only get educations/dowries IF there's enough money after the eldest son is provided for ... and the Lucases barely enough for the eldest son and the family he will one day have. If Charlotte even has a dowry, it's smaller than Elizabeth's, and that's a big if.
Charlotte has no money of her own, nor any prospect of ever inheriting it. While her parents are alive, they will support her. After that, her brothers will (and possibly her sisters, if they marry well). But that doesn't mean they'll be nice about it. "Unpaid servant/nurse and the subject of scorn" was pretty typical for how spinster sisters got treated. Think Anne Elliot from Persuasion, except that Anne actually had it pretty good as spinsters went: her father and sisters despised her, but there was enough money for them all to live comfortably on (even with Sir Walter's debts), and Lady Russell and the Musgroves like and respect her. Reread that bit up there from chapter 22 about the Lucas family reaction to her engagement, and ask yourself if Charlotte's treatment as a spinster sister would be as good as Anne's was ... and Charlotte has no Lady Russell to help her.
Later, when Charlotte talks to Elizabeth about it, she says: "I am not romantic, you know. I never was. I ask only a comfortable home."
Charlotte's goal, her grand ambition, is not wealth, it's not the Longbourn estate, it's a comfortable home. And she thinks that being shackled to Mr. Collins for the rest of her life is more likely to bring her that comfortable home than being a spinster living with her family.
I also think you're wrong about the book--and Elizabeth's!--ultimate conclusion on Charlotte. Elizabeth's initial reaction is very negative, yes ... but one of the book's major themes is Elizabeth learning that her initial reactions aren't always right, and that other people have different thoughts and needs and personalities than she does. The book doesn't admire Charlotte for her decisions, but it does have compassion for her situation.
That this combination of self-effacement (because she's willing to kowtow to fundamentally worthless people) and arrogance (because she fundamentally doesn't care about them, and thus doesn't mind that they impose on her) is a pretty shitty way to be.
Where are you getting the arrogance from? There's never an indication that she's arrogant or uncaring! Willing to kowtow to fundamentally worthless people, yes, but that's the social reality of her life. That was how society worked! You were supposed to dance attendance on your "betters" even if they're stupid, awful people. If you didn't suck up to them enough, there could be very real negative consequences. The book is not criticizing Charlotte, it's criticizing that aspect of society. (Which is a constant theme in Jane Austen's books, consider Sir Walter and Elizabeth's fawning over Lady Dalrymple in Persuasion.) Charlotte is in a lot more vulnerable position than Elizabeth is, financially and familialy.
If either of them are arrogant, it's Elizabeth. Elizabeth is the one who fundamentally doesn't care about Mr. Collins and Lady Catherine and thinks herself above them. Elizabeth is the one who's always mentally rolling her eyes at them and thinking about how awful they are. Charlotte shrugs and accepts them, and doesn't judge them, and urges Elizabeth to be less arrogant about them, and accepts the reality that they have power over her and her family.* She bows to the dictates she can't get out of, and arranges the household so that both she and Mr. Collins are happy. He's very happy, and she encourages him to do things he likes; the fact that those things take him away from home a lot benefits her as well, but it's not like she's hurting him or making him unhappy, she's literally encouraging him to do things he likes to do.
*Clergy, in those days, were basically appointed by the local richest person in the area. Mr. Collins is the parish priest at Hunsford because Lady Catherine personally hired him. She can't fire him! He's secure in that job for as long as he lives! She can make his life uncomfortable, but she can't get rid of him--that's why a particular parish is called "a living" for the priest in question. Clergy was also one of the most common professions for younger sons of gentry, because there were only a few respectable ones. BUT just being a clergyman didn't guarantee you an income; you needed a rich person to say "yeah, I like that guy, I want him as my priest" and give him the living. And the only people rich enough to do that that anyone in her family knows (so far as we know) is Lady Catherine. So by dancing attendance on Lady Catherine, she is also angling towards securing a decent future for one of her brothers. Clergymen didn't make a huge income, but they were comfortable if they had a parish of their own and weren't just a curate somewhere. Lady Catherine's patronage could easily be the difference between "comfort" and "poverty" for one of her brothers in the future.
If I'm not mistaken Elizabeth even observes that she and Charlotte can never be as close as they once were, now that she's gained a greater understanding of who her friend is and what she's willing to put up with for money.
It is true that Elizabeth observes that she and Charlotte can never be as close as they once were. But it's not that Elizabeth thinks Charlotte is greedy or arrogant or a bad person! It's that Elizabeth has realized they're actually a lot more different than she thought they were.
At the beginning of the book, Elizabeth's basic understanding of the world is that her perceptions of people are always right, and her understandings of situations are always correct, and that of course she knows better than everyone else. She's a sheltered 20 year old who is very intelligent, and like many smart 20 year olds hasn't really figured out that other people can see things differently, and have different needs and wants, without being bad or stupid. A lot of the book is about Elizabeth learning that. Charlotte is a good person and very intelligent, and Elizabeth likes her. Therefore, at the beginning of the book Elizabeth assumes that Charlotte must agree with Elizabeth on all fundamental matters because Elizabeth's perception of all people and situations is Right.
Prior to Charlotte's engagement, they have conversations where Charlotte quite plainly says that she disagrees with Elizabeth very fundamentally about marriage and courtship, and Elizabeth brushes it off and assumes she doesn't really mean it. After Charlotte's marriage, Elizabeth has to grapple with the fact that Charlotte was being honest with her the whole time, Elizabeth just wasn't listening because she didn't want to hear it. This doesn't mean that Charlotte is bad or greedy! It just means that she and Elizabeth are different. And Elizabeth has, up until this point, refused to see it.
One of the interesting things about the book, structurally, is that her reassessment of Charlotte--the realization that she misunderstood her friend in a lot of ways--happens at roughly the same time as her reassessment of Darcy. She has thought worse of Darcy than he deserved; she has thought she and Charlotte were more alike than they actually are. While she first realizes that she and Charlotte are different when the engagement is announced, she doesn't really get to see and understand the difference until she's staying with her for a couple of months in Hunsford. Seeing her with Mr. Collins and Lady Catherine and her own household, Elizabeth learns how Charlotte can be content with her life (even if it's not one Elizabeth could stand), and thus why she thinks the way she does. And realizes it's not wrong, just different. And then, towards the end of her stay, Darcy proposes and writes the letter and she learns she was wrong about him, too.
But while her realizing she was wrong about Charlotte leads to their relationship weakening, realizing she was wrong about Darcy leads to their relationship starting and growing to the point where they can marry. In some ways, learning to see Charlotte more accurately helps her learn to see Darcy more accurately.
