This is mainly an effect of Google being such a dominant player though. The sort of search results that are dominating Google is essentially a sort of local minimum in the fitness function.
It's pretty easy to create an search engine that outperforms google in dealing with SEO by valuing different things than Google. But as long as you have a system with a strong monopoly in search directing a huge swath of web traffic, there will be a sort of evolutionary pressure in the direction of what we're seeing on Google.
It's not a function of the sort of search enginge Google is, but its market position, and the fact that there is a metric shit-ton of money to be made in successfully adapting to its algorithms.
But this doesn't negate what I've said. Google no longer cares what results appear in front of you below the ads, unless those results are verboten topics, in which case they're hidden.
I use Marginalia reasonably often and quite like it BTW. It's a good tool. But there's a reason most of the popular alternatives to google are metasearch engines, it's because outperforming google is hard. So google threw in the towel, gave up competing on quality and now simply buys their way into every web connected product as the default interface to information to ensure hegemony. Marginalia succeeds at sifting through the SEO for the reasons you said, but then comes the problem of actual search, and even with Google being the big gas giant attracting all the garbage, nobody competes with peak google in quality, for a variety of reasons.
There is an aspect of cannibalization, Google can't really grow much more than they already have in the search niche and most other things they've tried (cloud, streaming video, social media, shopping, music) has competent competitors. The only way to present black numbers to the investors is to essentially milk the cow harder in a way that makes their offering worse.
I do agree that what has allowed Google to stick around is indeed that they're very well rounded. There are many competitors that do one or a few things better, but there isn't really a compelling replacement.
My point though is that most of Google's problems with "SEO" stem from their monopolistic position in the market. Their search engine is essentially a firehose of cash flowing into whatever ranks well. This creates a sort of dynamic where what ranks well gets more money, and is able to proliferate; and what ranks poorly dies off. This is essentially an evolutionary process. What is perceived as adversarial SEO is essentially Google's own shadow. It takes that shape it does because of the things they value.
Search engine spam is not just a flaw in Google's algorithms, it is a consequence of their position in the market.
Whatever replaces google, whether it's some sort of AI or Marginalia Search (not very likely) will have exactly the same problem as long as it exerts the same dominance over the web's traffic.
It's pretty easy to create an search engine that outperforms google in dealing with SEO by valuing different things than Google. But as long as you have a system with a strong monopoly in search directing a huge swath of web traffic, there will be a sort of evolutionary pressure in the direction of what we're seeing on Google.
It's not a function of the sort of search enginge Google is, but its market position, and the fact that there is a metric shit-ton of money to be made in successfully adapting to its algorithms.