Beyond the "You can write Fortran in any language" argument about properly structuring your code, there is a certain amount of truth in this. But in my opinion, this is mostly about familiarity. It's like a human language that uses another alphabet. Sure, if your native language is English, Polish looks more readable than Russian, as the latter is using a different alphabet. But that's a literally superficial point of view (cf. Lisp's parens), a stepping stone that is easily surmounted and doesn't change the total learning curve a lot.
English does look simpler than French, too. On the other hand, once you get beyond the accents, you know how to pronounce French, whereas that's not the case with English (cf. "ghoti"). Personally, I never was that bothered by e.g. the type characters, and quite often they were quite helpful about the context expected/required. But then again, I'm German, so maybe growing up with a somewhat ridiculous grammar and Funky Capitalization helped (there's a Perl module that allows you to write code in Latin, and here the cases and other grammatical structures replace the funky characters).
Perl certainly has its weaknesses. Some operations should probably be bound to types/objects instead of C-like functions, references are often difficult to untangle and the default object system ain't that grand (on the other hand: Moose). But I've yet to find a language that doesn't have similar weaknesses. Due to the somewhat funky syntax, they're just quite obvious. What does that dollar sign right there indicate? On what default variable is this code operating? Schwartzian what?
But I would call that "traceability" issues. And this certainly also happens with macros, meta-programming, complicated object hierarchies, generators, decorators, DoI containers, etc.
It probably does matter for casual programmers, i.e. people who don't work 20+ hours/week in Perl. Sysadmins come to mind. But if your involvement with the language is beyond that, I think "readability" is quite often a matter of taste, not more. Which, of course, doesn't totally disqualify this argument. It's just more a matter of rap vs. punk, not junk food vs. rucola salad.
The word itself is basically a joke, sure, but it serves as a nice accumulation of letters that have a somewhat unexpected pronunciation, certainly a feature of the English language. Pronunciation and emphasis are pretty hard for non-native speakers, and the written form doesn't help that much. On the other hand of the Latin alphabet spectrum, there are languages like Vietnamese, where you've got a boatload of accents and diacritical marks to help with it.
Although I'm not sure whether describing Perl as the Vietnamese of programming languages would be a good selling point, so I'll stick with French ;)
You also have to take the rest of the sentence into account too. English has a lot of words which mean different things depending on how they're pronounced (or, from the other side, that you can't know which way to pronounce until you already know the meaning), e.g. bass, close, desert, does, dove, familiar, have, intimate, invalid, lead, lives, number, object, present, produce, read, refuse, sewer, sow, subject, tear, wind, wound.