No. You’re dangerously wrong and this attitude gets cyclists killed.
I don’t cycle anymore, but I was a serious competetive road cyclist as a teenager. I competed in nationals several years in a row, and swept my state championships multiple years. This is an area where I am knowledgeable, and road safety is something I continue to be passionate about.
Here is the most critical piece of information: Bicycles are considered vehicles and have as much legal right to use the roads as cars do.
Would you be saying this if you were held up for work by a snow plow? By a slow moving tractor? By a truck with an oversized load? My single lane, tree-lined, winding country road is used by amish horse and buggy drivers, should they be disallowed?
Let me repeat that: Bicycles have as much right to the road as motor vehicles.
Just because other options exist, does not mean they are all equally good. Many larger roads are more dangerous, and large groups of cyclists can become a danger to other groups using a bike path, ASSUMING you have paved bike paths and not trails. Also, the speed limit on most bike paths is 16mph, and with dogs and toddlers effective safe speeds are often slower. When I was training for races, that was simply inadequate for everything but my recovery/low aerobic days. Yes, seriously. People whipping through bike paths at 18+mph are a serious danger.
A lot of thought goes into choosing an appropriate and safe route for riding. And yes, there are cyclists who do obnoxious or dangerous things, but the majority of the time the problem is the drivers not considering the possibility of other types of vehicles being out.
Cars have a sense of entitlement to the roads, but they are not yours exclusively.
Seriously, look in your state driving handbook.
If cyclists are routinely making you late for work, leave earlier. That’s it.