Good question! No. I'm a freediver and a paddling instructor, so I spend a lot of time wild swimming (sans life jackets) and in boats (wearing life jackets).
Life jackets are important when:
you're on the water in conditions where it would be unsafe to get in the water
there's a chance of entering the water unintentionally/accidentally.
Plenty of people boat & paddle in conditions unsafe for swimming (myself included). Cold water, strong currents, rough & windy weather, and being far from shore are all conditions that make swimming unsafe.
If anything happens and you go into the water, a life jacket will protect you in these conditions (even cold water, by allowing you to preserve body heat)
But what about when you're out in safe swimming conditions? What if you even plan to take the boat out to go swimming?
The risk is ending up in the water unintentionally.
If you're in a large or medium boat, on a paddle board, in a kayak, etc., and you end go in the water without meaning to, you've ended up in the water by means that put you at risk (capsizing, collision, sinking, fire, even just falling).
Injury, shock, etc. can prevent you from swimming, even if you're a strong swimmer. If the water is cold enough, you might experience cold water shock and a dangerous reflex called the cold water gasp. When you hit the water, you gasp automatically. The risk is inhaling a huge amount of water as soon as you go in, inducing panic, increasing the shock response, and increasing your risk of drowning.
How cold is cold water? Some people experience cold water shock symptoms in water as 'warm' as 14 degrees C (57 F). Which you'll find in plenty of popular lake and ocean swimming sites, even in the middle of summer.
The life jacket will save you if your ability to swim is compromised. It can even keep your head above water when you go in, period, keeping you warmer and preventing you from inhaling a bunch of water.
When you go swimming, you get into the water with intention, so the risk of being compromised is pretty low, unless you're adding an activity like cliff diving or something (or you're consuming substances that impair your ability to swim). So no, you don't need to wear a life jacket swimming if you know how to swim. You accept some level of inherent risk when you swim, like you do plenty of other activities.
Final note: having a life jacket near you is next to worthless. It needs to be on you and fastened. Are you going to grab and fasten your life jacket as you capsize? As you fall overboard? As you flee a boat that's on fire? Put it on and fasten it.
