I think most people, even within the queer community, fail to recognize trans men as an entity in itself. You're either trans or a man.
The idea that someone can be fully and wholly a man and still not be afforded the privileges given to cis men is foreign to most. The identity of "man" is synonymous with privilege and power, so when trans men say "hey I'm not as privileged as you think I am, there's more to it" that's unbelievable to most. They think we're purposely misgendering ourselves to gain imaginary oppression points. They compare us to cis men who refuse to acknowledge the privileges they do have because there's no such thing as a man that isn't privileged.
Its upsetting because it puts us in a bind. Either we suck it up and take the very real oppression we face head on and keep it to ourselves, or we advocate for ourselves and face ridicule for implying we aren't privileged or we're outright misgendered.
People truly do have a difficult time conceptualizing trans men. That's why other groups of marginalized men or intra-group male privilege are always brought up in these conversations. Not that those conversations are irrelevant, but it does mean they often get used as stand-ins instead of engaging with trans men directly.
Unlike other marginalized men, trans men are in fact marginalized specifically for being men. Because our existence disrupts the assumption that masculinity is natural, exclusive, or biologically assigned. The hostility directed at trans men is often rooted in the belief that we are “betraying” womanhood, aspiring to something we supposedly should not be allowed to become. That is gendered oppression.
Saying "men don't experience misogyny" or "men aren't marginalized for their gender" or "nobody is trying to stop you from being a man" is no different than saying "men can't get pregnant." Trans men aren't misgendering themselves, you just don't actually understand or respect trans people.
Perhaps people think they're being radically accepting of trans people but they're actually working from an incredibly cis-centric framework based on oppositional sexism.
