I think something that's missing in the "do trans men have male privilege" discourse is that some trans men *do* have male privilege but this privilege is depending on passing and passing is depending on being able to access and afford medical transition, which is related to both race and class.
"Trans men have male privilege" assumes that trans men universally:
1. Live in countries where medical transition is legal, affordable and accessible,
2. That they, specifically, are able to go through the slow and expensive process of getting HRT and other forms of gender affirming care prescribed.
Which is simply untrue. Most trans men do not have this privilege. Most of trans men are either forced to live as women or are otherwise perceived to be women by society and are therefore materially affected by misogyny, regardless of identity. Because misogynists do not care about how we identify, they care about power and maintaining their own place in the gender hierarchy, and that means putting anyone who doesn't fit their definition of what a man is down.