I agree that it's harder for Google, because we don't have the luxury of hard coding rankings, but I take big issue with this:
> So it's a tough situation for Google, especially when you factor into account potential impact on all the advertising revenue they're generating from these spam sites.
Adsense revenue does not enter in to ranking decisions. Our evaluation process is far from perfect, but revenue is not part of it. We're not blind to the fact that a lot of scummy sites run adsense, but the even scummier ones have already been kicked out of adsense and now use other ad networks or affiliate programs. "Denying spammers revenue" has been at times the explicit goal of projects that launched.
The tough situation I was talking about is that the net result of investing all this engineering effort to keep up with Duck Duck Go and Blekko still is likely to wind up as a revenue loss for Google.
In terms of AdSense revenue influencing search results, it would be great if Google published their ranking algorithms and implementations so that people could verify it for themselves. [Ditto for Microsoft of course.] But okay, I'll take your word for it that it's not factored explicitly into the ranking calculations and that you've done the analysis to make sure that it doesn't indirectly influence calculations. Even so, it may have affected resource decisions. At the organizational level, did the specter of losing tens of millions in ad revenue had something to do with why Google waited so long to start to address the problem?
> Even so, it may have affected resource decisions. At the organizational level, did the specter of losing tens of millions in ad revenue had something to do with why Google waited so long to start to address the problem?
At the organizational level, Google is essentially chaos. In search quality in particular, once you've demonstrated that you can do useful stuff on your own, you're pretty much free to work on whatever you think is important. I don't think there's even a mechanism for shifting priorities like that.
We've been working on this issue for a long time, and made some progress. These efforts started long before the recent spat of news articles. I've personally been working on it for over a year. The central issue is that it's very difficult to make changes that sacrifice "on-topic-ness" for "good-ness" that don't make the results in general worse. You can expect some big changes here very shortly though.
that's really the problem. Adsense doesn't factor into it... but it should. negatively. truth is, Google has created a superb "signal" that a website is crummy: Adsense. it would be funny if it wasn't so sad. and they can't do anything about it, since it would hurt the bottom line.