I don't think that's entirely true. There is a whole world of bad pronunciation out there, even from foreigners who learned the language and live in Japan. Plus, even if people are aware of the Japanese syllables, different nationalities add all sorts of inflections and word stress that make it almost painful to listen to.
Also, the "fu" (ふ) sound is a bit problematic for English speakers at least. While I would pronounce it with a hard "f" the way most English speakers would, it's actually a cross between fu/hu/wu (i guess the closest would be a soft "who") - the "f" part is very soft and as a non-Japanese speaker I am very aware that I don't quite pronounce correctly, but do my best. Even for my students, any word or name that is spelt with "fu" (like Fukuoka, or Fuka) will often be mis-spelled as Hukuoka or Huka.
I realize that my assertion may be considered controversial by many, but I stand by it. And that's mostly because I studied phonetics and phonology as a linguistics student. I will freely admit that some phones are quite difficult to nail accurately, but none of the really tricky ones exist in Japanese. The "fu" (ふ) sound can be achieved easily enough by just pursing the lips and bringing them together until a slight fricative is detected. In some Spanish dialects, a voiced equivalent of this sound can be found, as is often heard in the pronunciation of "vaca" (cow). So there's nothing really unique about the Japanese ふ. Your examples remind me of an experience I had when visiting Japan. I was at a restaurant and a waitress asked me, in English, if I'd like a 'hoke'. I looked at her, totally puzzled, until she mimicked the scooping motion one might make with a fork. After it dawned on me what she meant, I politely declined, explaining that 'hashi' (chopstics) were fine.
Pronunciation accuracy is just as much a mental activity as it is a physical one. It requires that the student relinquish all the habitual sounds one makes and instead make only those that are used by native speakers. Having an accurate IPA transcription helps tremendously when one is listening to speech because sometimes the native speaker may slur passages or speak them too rapidly. And then there's the whole elision business that happens with rapid familiar speech. As in English "I don't want to" being conflated down to 'aiowah?a'. (The question mark represents a glottal stop) Elisions can be shown in the transcriptions, of course. It all depends on how accurate you want to make your transcription. Beginning students in phonetics tend to transcribe spoken passages as if they were typing them, spelling each word in IPA as if it were perfectly enunciated by someone with no accents or dialectical subtleties. They're not really listening to the speech patterns. Instead they're writing out what they think they're hearing, not what they're actually hearing. It's when you get to the point to where you can accurately transcribe speech acts as they are spoken that you'll find you'll also be able to develop your own speech patterns to the level where you can at least sound reasonably close to native speakers. This is becausee you've developed your ear to the point where you can finally hear these speech patterns. And if you can hear them, usually you can speak them, especially with practice.
I must have succeeded at this at least somewhat because I often cringe when I hear Westerners, especially Americans, attempt to speak Japanese. It makes me want to give them a good shake and tell them to listen more carefully to the way the language is being spoken.
