Also the idea that there are people "curating" the way my search works is... unsettling I guess, they're deciding based on arbitrary criteria whether or not they think a user wants a site. What if I like the content DM produce?
> So it's a tough situation for Google, especially when you factor into account potential impact on all the advertising revenue they're generating from these spam sites.
Adsense revenue does not enter in to ranking decisions. Our evaluation process is far from perfect, but revenue is not part of it. We're not blind to the fact that a lot of scummy sites run adsense, but the even scummier ones have already been kicked out of adsense and now use other ad networks or affiliate programs. "Denying spammers revenue" has been at times the explicit goal of projects that launched.
In terms of AdSense revenue influencing search results, it would be great if Google published their ranking algorithms and implementations so that people could verify it for themselves. [Ditto for Microsoft of course.] But okay, I'll take your word for it that it's not factored explicitly into the ranking calculations and that you've done the analysis to make sure that it doesn't indirectly influence calculations. Even so, it may have affected resource decisions. At the organizational level, did the specter of losing tens of millions in ad revenue had something to do with why Google waited so long to start to address the problem?
At the organizational level, Google is essentially chaos. In search quality in particular, once you've demonstrated that you can do useful stuff on your own, you're pretty much free to work on whatever you think is important. I don't think there's even a mechanism for shifting priorities like that.
We've been working on this issue for a long time, and made some progress. These efforts started long before the recent spat of news articles. I've personally been working on it for over a year. The central issue is that it's very difficult to make changes that sacrifice "on-topic-ness" for "good-ness" that don't make the results in general worse. You can expect some big changes here very shortly though.
Conversely, "Do good (and minimize evil)" scales positively with company size and success. But it's nowhere near as catchy.