My accent has the cot-caught merger. If you don't have it, then you probably say "caught" with an open-mid back rounded vowel (/ɔ/).
But I say "caught" with the same vowel as "cot": an open back unrounded vowel (/ɑ/).
From my perspective, the vowel sound /ɔ/ just doesn't need to exist in English. I never say it. It's perfectly normal to me for "cot" and "caught" to be homophones. I can differentiate between such words just fine based on context.
But the pin-pen merger? No way! You need to differentiate those sounds! You can't just rely on context for that!
At least, that's my instinctive reaction, based on no facts whatsoever. But since the pin-pen merger does exist in some accents, I guess they must be doing fine without the distinction. Somehow.
