Sure, but Google has two kinds of customers: searchers and websites.
Maybe you believe websites shouldn't hide from Google because it makes no strategic sense. Twitter and Facebook don't let Google index their content. Why would they let Google front-run ads on their content? It's not a catchall rule, to hide with robots.txt.
Maybe websites should just pay up: they should auction for higher results, if their business depends on search placement. But that's misunderstanding Google's role.
Google isn't a salesperson. It's more like a road. Both bring business to your door (literally), but the latter gets used by everyone for everything. And it turns out it doesn't matter how you pave the road (roads have been public since the Egyptian times, yet road-building technology has changed dramatically) or whatever.
To Nextag, Google is a utility. It is only by tradition that some monopolists become utilities and others do not. How do you regulate Google for what it is: a public utility?
It's no wonder that Google cracked down; they were destroying its main service - search - in order to make a cheap buck for themselves by driving out useful results and driving searchers to turn to them in desperation.
Isn't that exactly what AdWords is?
As a consumer, I definitely what to know what is being paid for to show up and what isn't.
Google has to get searchers to sell to advertisers. That said, the generic 'websites' mentioned above are not Google customers. Google does not "owe" them anything – they owe the searchers the best results and the advertisers clicks at a certain cost. The only thing those other websites do with their free listings are attract eyeballs to Google's engine.
The entire reason Google improves the search engine is to keep you coming back so they can learn more about you to make their advertising even better. You, as a user of Google, are their product. They are selling YOU to advertisers.
Google is a company, not a utility, not a public service. They are in the business of profit.
However, cheap polemics (like being very adamant about who gets to be the customer) help nobody. Two-sided markets can push prices for one side of the market below zero, that’s just an inherent property of those markets. That does not mean the one side is the product.
Google has to provide value to you, otherwise they couldn’t make money. If they screw you they screw themselves.
Now, this doesn’t mean that Google can do no evil, but it does mean that you can’t just assume by default that the relationship between users and Google is an exploitative one.
(And two-sided markets are nothing new. Google has essentially the same business model as the free magazine that informs me about cultural events in my hometown. Because of their very well targeted ads – local restaurants, businesses, etc., often pointing to current events – they can offer that magazine for free. That doesn’t mean I’m the product.)
It's also useful to recognize that for some of Google's lines of business, the payment offered is not money, but attention (or perhaps habit). Then, searchers are again customers, after a fashion.
This better recognizes the searchers' customer-like power, in some dimensions. Otherwise, as just the 'product', were they mined from the ground? Grown like crops? Manufactured? No, their attention was acquired in a voluntary, market-like but non-monetary exchange of value for value.
That said, the greatest search engine in the world isn't that great if no one uses it. Search volume (and advertising platform quality) is what drives advertisers to Google. If their results suck, users will use something else and lose the product that Google needs to be able to sell to its customers.
(You can say the same thing about the rest of their products as it is all about collecting peripheral demographic information that allows advertisers to directly target them on the content network).