Short of subverting human nature of everyone on the Internet it will never be possible to have a leading search engine that people aren't constantly trying to game.
The best we can hope for is gaming behaviors to roughly align with desirable search results.
I don't see an ethical disconnect. Google is the world's largest scraper, making almost all of their revenue by serving ads next to the content of everyone else but them, yet there is an "ethical disconnect" by voluntarily choosing to still abide by their guidelines?
> Google is the world's largest scraper, making almost all of their
> revenue by serving ads next to the content of everyone else but them,
That's oversimplifying the service and value Google offers. This is the typical passive-aggressive SEO stance. Deriding Google for the service it offers, but at the same time craving it's attention. Because you know, Google gives you a steady stream of customers.
People use Google because it helps get them to the websites that offer the content that people are looking for. It's (relatively) good at matching seekers with the information they seek. That's what brings people back to Google over and over.
And so, the page they deliver has value to people.
Yes, they scrape content from everyone else. Including themselves (e.g. a search for "Google webmaster guidelines" returns support.google.com).
They also add to that their special ingredients - algorithms that dissect pages, classifying, inferring meaning, deriving synonyms, inferring intent. Weighing, scoring, evaluating every page out there. So when a customer comes a-querying, it uses those calculated metrics to decide which 10 pages get to be mentioned on a search results page, and what snippet/extract of those 10 pages to show.
The value Google adds is the selection of which pages to link to based on the query and what it knows/infers about the pages themselves. That is no trivial thing.
It's nowhere near passive aggressive, just a statement of what Google does. Google does give a steady stream of traffic, but it's my goal to diversify traffic streams as much as possible at the same time - Google's becoming less important as a direct source of traffic from organic SERPs.
I'm not saying they don't add value, I'm saying that their entire business model is built on the back of the world's content.
I'm not sure why HN hates SEO so much, since Google doesn't...they hate SPAM, not SEO.
The ethical disconnect is painful.