In my experience, another big part of the problem with talking about "realism" in fictional milieux is the product of a specific Type of Guy employing the term as a sort of semantic bait and switch, sometimes without consciously realising that this is what they're doing.
There's a particular recurring discussion of "realism" in media that goes something like this: "okay, but realistically the heroes would always win because they'd just shoot the villain while they're monologuing" – while refusing to acknowledge the obvious follow-up question: "wait, but if monologuing reliably gets you shot, where do all the monologuing villains come from?"
i.e., what we're really discussing is not a milieu which has adhered to some notional model of "realism" ab initio, but one which was apparently governed by the conventional tropes of its genre right up until the moment the character the person framing the scenario wants to win walks into the room, whereupon "realism" asserts itself.
Heck, there were folks doing this song and dance in the notes of the post this one is following up on, trotting out hypotheticals like "in a realistic fantasy setting the twelve-year-old chosen one would always lose because experience trumps skill and the power of friendship isn't real", implicitly taking it as given that in a milieu where this is true, people would still be handing out magic swords to twelve-year-olds.
It's basically treating those silly "How [Media] Really Should Have Ended" YouTube videos as a legitimate critical lens, and in circles where this song and dance is common, it leads to a lot of people reflexively shutting down the moment they hear the word "realism" because they assume (often quite reasonably!) that oh great, it's That Guy again.