Who wins in a fight? Ultra Instinct Shaggy Rogers or Big the Cat after someone's been mean to his friends?
Well, let's break things down, I guess. Where did Ultra Instinct Shaggy start? Scooby-Doo and the Phantosaur, right?
If I remember right, the setup for this movie is that years of being in Mystery Inc. have fried Shaggy's brain. He's a nervous wreck, riddled with anxiety, and he can't stop having panic attacks. He gets hypnotized into forgetting his anxiety and the biker here says his trigger word, making him a paragon of confidence.
So, really, this is just normal Shaggy, just he's ready to throw down. Instead of the years spent chasing monsters making him afraid of everything, it now makes him prepared for anything. He easily handles an entire diner full of angry bikers like it was nothing.
Big the Cat is a little harder to estimate. Most people seem to be afraid at the sheer idea of having to fight Big. He doesn't really seem to get into many scrapes. Most of what we've seen of Big fighting comes from the Archie comics, where he does things like catch and throw Mecha Sonic like it was nothing:
He also punched Drago through a wall in a single swing:
And similarly eviscerated Metal Amy Rose:
By that measure, I think Big is significantly stronger than Shaggy. If Big can punch a hole through several layers of machinery and armor, he could kill Shaggy with little effort.
Now, you did say Ultra Instinct Shaggy, but what is Ultra Instinct in Dragon Ball? It's a state of mind more than it is some kind of powered up form. It's about clearing your mind of all distractions, shutting off all external stimuli except the pure essence of fighting. It's like, in a science fiction space opera, right, where they shut off all of a ship's systems and divert power to shields. You're shutting off your brain so you don't care about family, friends, love, even basic concepts like eating and sleeping. You completely empty your head and get so locked in, so focused, so connected to your Ki, that you can basically react to what your enemy is going to do before they even do it.
Ultra Instinct does not necessarily give you more strength, it just makes you the best version of yourself. And for Goku, he had been training with the best of the best across earth, the galaxy, and the afterlife. Ultra Instinct is focusing Goku down to unleash every ounce of his potential. That's kind of the ultimate "message" of Dragon Ball Super: that strength only goes so far, and it's knowing how to use that strength that matters most.
So let's say Shaggy taps in to the real, actual Ultra Instinct. There's precedent. Scooby-Doo and the Samurai Sword puts forth a similar claim: that there's a state that can only be achieved by clearing your mind. Here, Scooby ultimately deals the final blow, but it at least puts it on Shaggy's radar. (The joke probably being the same: both Goku and Scooby are so dumb it's easy for them to clear their minds like this)
But even considering that, I think Big still wins this. Shaggy has faced a lot of monsters in his day, but Big has survived multiple world-ending (even reality-ending) apocalypses. Shaggy also seems to have an extremely high metabolism, repeatedly consuming more than his own weight in food throughout the day, even at rest. Any advantage he'd have is short lived, whereas nothing about Big suggests his strength only exists in short bursts. Even the ultimate, most focused version of Shaggy isn't going to last forever against that mountain of a cat.
