A lot of the suspicion towards Loki in Thor 1 stems from the other characters not knowing jack shit about magic or what Loki's capable of, and assuming he can do pretty much anything and overestimating his abilities. It's like blaming an IT specialist for an intricate cyber-attack because he's the only IT specialist you know, or your parents thinking you can hack the US government because you reset their modem. Granted, Loki was behind a lot of stuff in this movie, but that's just how it feels and that's hilarious? Especially because a random enemy's words are enough to make them suspect one of the crown princes?
How common are Loki's abilities? I assume hiding from Heimdall's sight is very rare for my own sanity. But it's weird to assume only Loki can do that EVER.
But are the Warriors 3 and Sif really assuming Loki can unbanish Thor and give him back his powers? I strongly believe he can't (at least in Thor 1, because he gets some power-up in the T:TDW ending that Odin manages to undo (implying Odin's magic is much stronger)).
And if we're talking about the comics, the Odin-force is specific to Odin, and is passed onto Thor. I wonder if they're assuming he has inherited the Odin-force, but they're unaware of his Jotun heritage and that he hasn't gotten it? And of course, Loki can't tell them why he doesn't have it... This is just me, extrapolating and making shit up.
Thor 1 establishes a very imbalanced (literal) power dynamic between Odin and the rest of the family. Odin's banishment powers are Odin-specific. Frigga in the deleted scene, would have undone them if she could. Loki would have been able to gain Mjolnir's power by changing the parameters of worthiness if he could undo it.
Either Loki was always treacherous and terrible but incredibly powerful, like in the comics (after IW and sequels, I am not inclined to believe this) or Loki was actually decent/trustworthy for most of his life but his magic has limitations.