*rubs together my terrible linguist hands* hahahaha yes!!! sorry to bother you both on this months-old post, but i have some things to add!!
in my opinion, burgeoning-ambitionさん is totally right when he says what we're interested in here is probably the distinction between phonemes in japanese. in phonetics we have a joke: are phonemes even real?? we say this bc once you start looking at the deep, nitty-gritty MATH of phonemes, the boundaries between them can really start to blur in fascinating yet panic-inducing ways (lol). in western linguistics, we represent phonemes with ipa symbols as a shorthand, because it's convenient and legible, but if we forget that the ipa is in fact a shorthand, we also forget that phonemes are not sounds but representations of groups of sounds.
vowels make a good example of this. have you heard how modern standard arabic "only has three vowels"? this is a common vowel inventory chart for arabic:
if you're not careful, it's really easy to look at this chart and say, damn, arabic has only got three phones (sounds) in its vowel inventory. but i think we can all be a little bit less credulous than that. what's really going on here is probably a big venn diagram-type situation whose outermost points end up getting called "phonemes". like this:
!! this is a great big simplification i made in 30 seconds !! i am not an arabic/semitic expert !!
if this is in fact what the vowel inventory looks like in msa, then probably, msa speakers have just as much vowel space as speakers of any other language. it's just that what acoustically counts as /i/ is way more lenient in msa than in many other languages. in broad terms: the acoustic boundaries of any given "phoneme" are (probably) language-specific.
at this point i hear you say: what does this have to do with japanese?? let's return to that!
first of all, i want to share my own observations and experiences. i have observed this: japanese speakers can produce both [r] and [l]. in regular speech, [r] tends to come out as a flap or tap, but if you get into singing and other registers of speech, [l] surfaces more frequently. so, as opさん asks, why is [l] not its own phoneme in japanese?
honestly, i can only speculate as to why, originally, /l/ did not enter the japanese inventory, but since rhotics are cross-linguistically extremely difficult to acquire (in your L1!), liquids as a bigger category might fit that trend. that is to say, /r/ and /l/ are literally just harder to produce, so some languages leave them out of the inventory.
however, what we can take a good guess at is the size/boundaries of japanese /r/. think about a typical "mistake" a japanese speaker might make in english: she might confuse [r] and [l], like not being able to hear or produce the difference between "right" vs. "light". this is not the same as not being able to produce "light" at all! why should someone confuse two phones in their L2 which, under the right circumstances, they can produce in their L1? because the phoneme boundaries are different. [l] exists in japanese; speakers just analyze it as a member of /r/. this is exactly what burgeoning-ambitionさん said!
[f] and [v] are trickier. i will mention another personal observation here: in my experience, japanese speakers do not actually produce ヴ with [v] and do not actually produce ファ, etc. with [f] or even [ɸ]. to my ear, the consonants in these new kana tend to come out much closer to [b] and [h], respectively. so i would hesitate to say [f] or [v] have entered the consonant inventory, even in loanwords, and i would also hesitate to say /ɸ/ has become phonemic in japanese. even if a speaker produces them, they're probably still firmly under the phonemes /b/ and /h/.
cross-linguistically, this is really interesting bc [f] and [v] are among the fricatives that tend to take over from even more difficult fricatives (like the interdentals), which is really stereotypically obvious in lots of dialects of british english for example ("bruv"). but in japanese, we can't even get that far! it might be that phones like [f] and [v] are much more accessible from above than below. in other words, if a language has "harder" sounds like the interdental fricatives, it probably has access to "easier" sounds like the bilabial fricatives. since japanese isn't at the level on the hierarchy where the bilabial fricatives are, it's really difficult to import them via loanwords, and the phones get naturalized. (btw does anyone remember which hierarchy this is?? i've been trying to think of the name of it for like five minutes and can't. message me if you know lol)
uhh anyways what did we learn?? possibly nothing lol, especially since burgeoning-ambitionさん covered a ton of this and the vast majority of it is just (well-informed) speculation. i guess i just thought i'd add some more details! let's all enjoy linguistics forever!!
source: wrote my master's thesis on japanese phonology! message me for actual sources/discussion if you want :)