A crucial part of the conversation around transandrophobia is the distinction between personal identity and sociopolitical categorization; or, to borrow one of my least favorite made-up grad school words, "positionality".
"Positionality" refers to where you exist in relation to power. It's "white" vs. "person of color" or other race-related classifications, it's gender, orientation, socioeconomic class, ability, "fat" vs. "thin", and whatever other axes of power you can identify.
"Personal identity" is how you choose to describe yourself; the words you prefer, the specific labels that make sense or feel right to you, and the words that may technically describe you, but that you don't want applied for any number of reasons.
The difference between these two things is why, for example, "bi" might technically describe pan people as well, but a lot of them don't actually identify as bi; and why, at the same time, a lot of statistics around bi positionality- the specific oppression that targets bi people- also apply to, and include, pan people.
The issue is a lot bigger than this, and there are a ton of arguments within the queer community based around the conflation of these two concepts in one way or another ("don't call it the queer community" comes to mind). But the one I'm interested in here is the way this impacts transmasc discourse, specifically.
There's a particular confusion that happens with trans people's identity vs. positionality in queer discourse. Cissexist society says that trans people's identities don't matter; that they do, and also should occupy the position of whatever gender they were assigned at birth. Position determines identity.
The common argument to this is to just flip it the other way around: trans people's identities do matter- and that those identities determine positionality.
This makes sense, to some degree:
Trans people face transphobia: identity = trans, therefore position = trans.
Trans women face misogyny: identity = women, therefore position = woman.
Trans men identify as men: identity = man, therefore position = man.
"Tans male privilege" is the notion that trans men, upon identifying as men, instantly gain access to the position of maleness.
But this is easy to poke holes in; telling someone you identify as a man doesn't stop them from seeing you as a woman, and that's kind of a vital function of transphobia in the first place. In fact, doing so would immediately subject you to transphobia.
So people think: okay, if trans men don't occupy the position of "man" because they identify that way, what does that say about trans women? Does that mean trans women do occupy the position of "man", despite identifying as "woman"? Does that mean they don't occupy the position of "woman" at all- and therefore cannot be subjected to misogyny?
Obviously, that's also not true! Trans women do experience misogyny; this is a well-documented fact.
And that adds to the confusion: if trans men experience misogyny, does that mean their position = woman? (And isn't that just what TERFs believe?)
The problem here is twofold:
We're still conflating identity with position- we're just arguing over which one determines the other.
We're ignoring that "trans" is itself a position. Trans people don't necessarily occupy the position of either binary gender; we are often just seen as "trans", and placed in that position.
This position is also a little bit unique in that it's particularly mobile: society doesn't want to acknowledge that this is a valid way to exist, and so the existence of the position is denied as much as possible. Trans people are, as a result, often categorized as "women" or "men" depending on what's convenient: if the transphobe in question can subject them to misogyny by categorizing them as a woman, or if they can paint them as "dangerous" by categorizing them as men.
If we can understand that position doesn't necessarily determine our identity- our actual gender- we can understand that trans women are both exactly as much women as cis women are, and that they occupy a very different position, and have very different experiences.
We can also understand, through these ideas, that trans men can identify as trans and as men, and that these identities don't necessarily determine position. Trans men don't occupy the position of "man", even though we identify as and are men.
We can also understand that trans men also aren't necessarily women, even if we do sometimes occupy the position of "woman", because every trans person can occupy that position if and when it's convenient to transphobia. The same is true of the "man" position and trans women.