TLDR: This got longer than I meant. The important takeaways are in bold, if you just read those you'll get the point.
I do not work in a sex toy shop but I've taken some upper level sex ed classes and I live with a soap maker who has some pretty impressive microbiology credentials. Unscented soap is absolutely fine to use on toys. In fact, if you have sensitive mucus membranes, a truly simple bar of soap (oils, fats, lye, pretty much nothing else) may be a better option than toy cleaner.
Even if soap residue remains (which it shouldn't, you should be rinsing thoroughly and letting your toy dry completely before putting it away! letting it dry completely is a big part of what prevents bacteria growth) it will not increase the amount of bacteria that your toy can "hold onto." As for viruses. It's a lot of science but essentially, viruses aren't going to get trapped in soap residue. They might sit on top of it after it's completely dry, but even without the soap residue, viruses will still sit on top of your non-porous toys to the exact same extent. The same is also true of bacteria. Soap residue does not increase the risk or amount of bacteria or viruses being on your toys.
The risk of soap residue is that you might be sensitive to something in the soap and it may cause irritation. This is also true (in my experience as someone with super sensitive skin, is more true) of dedicated sex toy cleaners. In fact, I cannot use most sex toy cleaners because any residue left behind at all causes irritation.
The point of cleaning a sex toy is not to make it sterile and completely free of bacteria until your next use. Such a thing is not possible. Remember, bacteria is on everything, and prefers to live on surfaces. Even if you use a dedicated sex toy cleaner on your toys, even if you sterilize them in boiling water, when you take them out to use them again, they will have bacteria on them. Actually, they'll have bacteria on them even before you put them away.
This does not mean you don't need to wash your sex toys because they'll have bacteria on them anyway. You're washing your toys to eliminate the bacteria, viruses, and fungi (yep, fungi) that have gotten onto them while you've been using the toy.
I literally just lather my hands with the bar as if I were washing them, and then use my hands to clean the toy, rinse, let dry. Done. If it's been a while since I used the toy (and therefore a while since I last washed it) I will also wash it before use, but this isn't strictly necessary. Chances are good you keep your toys in the same area you live, though tucked away in a drawer or box. In which case, you've already been exposed to the bacteria currently hanging out on your toy.
