Because PHP (heavily inspired by Perl, but not as well thought out) came with all the glue to directly use it to generate web pages. Then came Ruby (also inspired by Perl, but with a more mainstream syntax, and nicer in some ways), and Ruby on Rails conquered that space even more. Also, Python came along which is indeed a bit more suitable for larger projects needing structure.
Meanwhile, Perl was being badmouthed by people who couldn't handle its flexibility and thought it was unreadable and unmaintainable. But that wasn't the main issue, after all PHP was worse in those two respects, yet it flourished.
Ruby is very cool as an easier, shinier, more ergonomic and elegant Perl but the lack of block-based lexical scoping with compile time checking (use strict + my, duplicated as "use strict" and let in ES6, much to my happiness when using javascript) and the fact that while Ruby's core OO is much easier to use than bless() it's horrifyingly limited when you're used to Moose/Moo means I've always bounced off it when I've tried to learn.
Perl's sheer flexibility means there's a gulf between baby perl and good complex perl during which the developer doing the learning tends to make a complete mess (I definitely did this for a while) and some people never get past that stage so a lot of pre-existing perl code is awful and much though I love perl myself the language is definitely at least partially to blame.
Many of us wrote our own 'php' in those times. Just some (fast)cgi blurp that read a vanilla perl file and wrapped it (via Apache pre-script/post-script) in the tooling to get the $_GET/$_POST, print "Content-Type: text/html\n\n", set/get cookies etc and returned the output of the script as something like XML. We stuck with Perl for a long time because of cpan and it was, especially then, a much better language when used in the right way. It was often abused (and I still do, because slightly complex scripts are still really easy to do in perl), but so was php. So I'm guessing the ease of php deployment if you didn't have access to the Apache of the server running would be a big thing.
I was there during the transition. Even rewrote a site from perl to PHP.
For newbies, not having have to write the "magic" string,
print "Content-Type: text/html\n\n";
before being able to output anything probably felt quite friendlier and being able to mix HTML into the code, though it's not a good practice.
And the PHP standard library has quite a few things built in and combination of MySQL as LAMP was the hot word then, so I guess that's how it got traction and perl was gone from the web development.
I do like the Unix heritage of perl though. It does make you feel like a hacker when it feels the language has the shell syntax half baked in.
And nobody starting web dev now with perl should even consider it. Modifying an existing script that otherwise 'just works' sure. In my various other posts here I've recommended Mojolicious.
You saying nobody starting web dev should use CGI.pm? Or CGI in general? Because CGI itself has its uses even today. It's much easier to understand and get off the ground than something like Mojo or Plack (I've used them all), and has its strengths compared to the others.
However, if you are saying no one should use CGI.pm, then I agree. That module is a bloated mess. Much more efficient ways to use CGI without that thing.
PHP is "worse" in these respects when compared to other mainstream languages, yes - we can probably grant that, although it's not as bad as it once was.
Perl is just one big game of code golf. There is absolutely an entire era of developers who looked at that and ran far away (and I do not blame them at all).
"one big game of code golf" would probably be a decision by the author, wouldn't it? I know a person or two who could fall into that category, but it's hardly the fault of the language.
No, this is specifically a comment on the syntactical choices often found in Perl codebases and the penchant for brevity that is common in that community.
I think the other problem was that Perl 6 came out and was totally incompatible with previous versions. Python did something similar with 2->3 and it caused huge problems.
That was later though. Interest in Perl was already declining a bit then. And the problem was more that the design and implementation of Perl 6 (now Raku) took extremely long, and attention to Perl 5 suffered from that, as Perlistas thought Perl 6 would be the Next Great Thing.
Not "almost", it is a completely new language with a passing ressemblance and IIRC was by and large advertised as such from the get go; the name "Perl" was kept as it was a spiritual successor, which they realised made it confusing so they changed it to Raku.
Perl 5 and 6/Raku have as much in common than Perl 5 and Ruby have.
IIRC was by and large advertised as such from the get go
It was not. For many years it was advertised as the next version of Perl (with the intent being perhaps a Perl 5.10 release and then no more major releases with the 5 version number), then years later a "sister language", and then finally now (but not retroactively) a "completely new language in the Perl family".
Very early (2000) the goal was for Perl 6 to be a rewrite[0]. There's no way this would have been a 5.x, especially given how the Perl 5 parser works and the class of warts that were aimed to be addressed:
> Perl 6 To Be Complete Rewrite (But Not What You Think)
> Perl 5 will not be abandoned, but will primarily be concerned with bugfixes both major and minor.
> The meeting for members of the perl5-porters mailing list was the result of an earlier, smaller meeting of Wall, Nathan Torkington, Chip Salzenberg, and others who basically decided that Perl needed to be fixed in certain ways, and that a rewrite was the best way to do it.
Quickly, being free from backwards compatibility gave room to try stuff out, and given the length of that design process, this resulted in features that they found out could be immediately useful if backported in some way to Perl 5:
> First, Perl 5 isn’t going anywhere. If anything, the rate of patches and changes to the code has increased. Cleanups from Ponie and the Phalanx project continue to improve the design and implementation, and new features from Perl 6 are making their way into Perl 5.
> Second, the opportunity to do the right thing without fear of breaking backwards compatibility opened up a lot of possibilities for impressive new features.
Granted, there's the following bit in the 2000 post:
> have a clear and automated migration path, which may include a backward compatibility mode
But back then (ca 2002-2005) I was only a Perl apprentice and a very junior developer yet it still struck me that the discussions highlighted differences that were expansive and fundamental enough that "next version of Perl" meant Perl 6 would be to Perl 5 what e.g Mac OS X† was to macOS 8/9 and (in retrospect since that happened only later) significantly different in class from e.g Python 2 to 3 or MRI to YARV.
That was my perception of it anyway; probably they weren't entirely clear what was the actual goal, hence a lot of bits were open to interpretation, and it got refined and bounced back and forth around a general baseline direction over 5-10 years.
Meanwhile, Perl was being badmouthed by people who couldn't handle its flexibility and thought it was unreadable and unmaintainable. But that wasn't the main issue, after all PHP was worse in those two respects, yet it flourished.