I think part of the reason for this is that web site developers have got out of the habit of optimizing for search engines. I'm often surprised by how self-contained the requirements for a website are now, even among otherwise technically sophisticated clients. There'll be a beautiful site in React that absolutely sucks for SEO, but no-one will mind because a) it's unclear how big an audience there should be for the site, and b) the "all your hits come from search engines" was broken ten or more years ago by social network linking, so the question of how you get an audience seems much more arbitrary, and less connected to google.com.
