If you define sexual violence as anything other than nonconsensual actions in the sexual sphere, you're going to be excusing and/or dismissing(the effect of) sexual violence that is not "bad enough" in your opinion, and/or attacking or enabling the attacks on people for consensual actions and things that do not involve any sentient beings other than those people themselves.
If you agree that sexual violence means nonconsensual actions in the sexual sphere but believe that is exclusive to the sexual sphere, you are Actively Excusing A Lot Of Violence and do sex exceptionalism on the literal highest level, and some of the people who do this also claim to be against sex exceptionalism.
For specifically the "weaponization of consent rhetoric" which some of you are trying to argue against and end up doing that, I am copying my Shirt Examples (https://www.tumblr.com/dyspunktional-leviathan/777176885489762305):
A comes in wearing a shirt. B doesn't like it / thinks it's immodest / whatever. B should shut the fuck up.
A comes in wearing a shirt. B's abuser worn a similar one and this triggers them. They may leave if they want&can; if they have to see A often, they may have a discussion about it. It is, in fact, possible and alright, that if that shirt does not matter as much to A and does heavily trigger B and they cannot/don't want to avoid seeing each other that A should not wear that shirt around B.
A comes in wearing a shirt. It fucks with B's sensory issues (like having bells on it or something or being an eye-straining color). I guess, pretty much the same as the above applies. And in these cases, if it is both important to A that they do wear that shirt and to B that they don't it will be a dealbreaker to them seeing each other.
A comes in wearing a shirt. It has writing on it personally demeaning B. Has B consented to that?
A comes in wearing a shirt with hate group imagery. A should be gotten the fuck out unless it's like, a scene / a theatre rehearsal / etc and they're in costume which is not an actual reflection of their beliefs, and everyone around has consented to seeing that costume.