Google Maps tells me the whole route, not just 'what to do next'. I can understand how the route was produced, and because it's shown on a map, I can see whether it matches my intentions before I follow it.
Amazon is less transparent, but they usually do show things like 'we recommended this because you bought that, and 40% of people who bought that liked this"
I'd hardly say that Facebook tells me who to I should be friends with - they mostly suggest people I might already be friends with to make it easier to connect with them. I don't know what algorithm they use, but I guess it's based on mutual friends and their scrape of my email box from ages ago. I wish they would tell me more.
Google on the other hand is completely secretive about how they generate their recommendations (i.e. the search results). How do I know the results I see are aligned with my interests?
Since Google's customers are advertisers, it wouldn't be unreasonable to guess that the search results are at least to some extent tuned to maximize advertising revenue for Google, as opposed to my needs for quality, breadth of coverage, historically significant results etc.
The algorithm may not be tuned to favor specific interests, but it clearly has a structural bias that favors certain kinds of information over other kinds.
I think that having so much of the web's information flow governed by secret algorithms is very much a cause for concern.
Gruber took Schmidt's quote out of WSJ context [http://online.wsj.com/article/SB1000142405274870490110457542...] before applying some sinister spin. The WSJ article conveys the idea that in the future, Google will be better at figuring out what information is relevant and unknown to you at any given moment (so much better that it can be proactive about delivering that info without annoying you).
I don't know why Gruber feels that Google is Apple's enemy, or why he's decided to attack Apple's enemies rather than praise/defend Apple's tech, products, and culture. But, many people seem to be worried about Google and not worried about Apple, despite the fact that the latter is much more secretive. I guess it's because Google seems more important, being involved with so much of "the web's information flow" and not just with making beautiful computers and accessories.
Of the references you list, only the PageRank one has anything to do with the search algorithm. It's a great paper. The backrub paper (http://infolab.stanford.edu/~backrub/google.html) is also great, and has been a personal inspiration, however that information pre-dates the founding of Google as a corporation, and the search algorithm been developed in complete secrecy for more than a decade since then.
There are more than 200 other signals used to determine relevance, and PageRank is only one of them (http://www.google.com/corporate/tech.html). The others are completely secret as far as I can tell, and there's no indication what weight is given to PageRank now.
I'm not attributing malice to the secrecy, but there's no way to know what biases are embodied in the algorithm, and I think that's a serious concern.
I don't think it's unreasonable to claim that Google is as important as the press now as a source of public information. The biases of the press are very much a concern for society, so I think we should be equally concerned about the potential for bias in the search results.
As for Gruber, my guess is that he feels that the discourse in the tech press/blogosphere has demonized Apple heavily for the closed nature of the App store, whereas Google is praised for its 'openness' with much less examination of what that really means.
Personally I think that Gruber can be childish about this, but I think both companies deserve rational criticism.
I'm more worried about Google than I am about Apple because their secrecy is about how public information is prioritized.
I do have similar concerns about how Apple prioritizes applications in the App Store, but for now that seems a lot less important than how Google prioritizes websites.
> Of the references you list, only the PageRank one has anything to do with the search algorithm.
...
> There are more than 200 other signals used to determine relevance, and PageRank is only one of them (http://www.google.com/corporate/tech.html). The others are completely secret as far as I can tell, and there's no indication what weight is given to PageRank now.
But, sure, Google has large scope and importance, and they do have secrets (about search rankings and other topics). Policy statements and economics papers they've published about the alignment of users' interests with their own are only suggestive and don't guarantee good behavior. As long as they have secrets we can't be sure they're not doing evil (or even just failing to serve each of us as well and faithfully as possible).
Thanks for pointing out those other links. One of them is about news, and not search, but the others do shed light on algorithms that might be part of how Google does ranking.
But as we agree, the actual algorithm is secret.
Incidentally, I'm less worried about deliberate evil than I am about systematic biases that we don't know about. Here on HN people show a lot of interest in cognitive biases and the importance of being aware of them. I think the biases of the 'extra-cognitive' systems we use are just as interesting.
Is any other search engine is or was transparent about their algorithm to your liking? Bing? Yahoo? DuckDuckGo?
If google published the definitive spec of their algorithm (i.e. source code to their search engine), would you read hundreds of thousands lines of code? Would you be able to comprehend it? Would you re-read it every week to make sure that hundreds of commits made by google engineers didn't change it in a way you don't like?
It's preposterous to think that you could comprehend their algorithm in a way that allows you to make an informed decision about whether to use or not. Let's assume you understand it and don't like how it works, what you gonna do? Use Bing?
You don't need reassurance that Google's algorithm is aligned with your interests. It isn't. Neither is Bing's algorithm or anyone else's. BMW doesn't make cars that are aligned with your interests. Apple doesn't make phones that are aligned with your interests. They all make products that are aligned with their interest to make money, they put them in the market and you decided with your money whether you think they serve you or not. That system works pretty well.
You have a choice to use any web search engine you like. You can decide whether you like their results or not by making queries and inspecting the results. You are free to create a competing search engine. Those are your rights.
You don't have a right to require google to disclose how their products work. There's a name for it: trade secret. It's protected by law and it's google's right to keep it to themselves.
I don't think I said I had a right to require google to disclose how their products work.
I do want reassurance that Google's algorithm is aligned with my interests. I want that from everyone who provides me with a service, and every corporation I've worked for has wanted that from their suppliers. If I have evidence that a company is going to work against me in the future, or is doing so now, then I'll choose not to buy their products. Understanding what you're buying is a key part of making markets performing well.
I think BMW does actually try to make cars that are aligned with their customers interests, that's why their customers pay for them them. One part of my point was that I'm not Google's customer - the advertisers are.
I can't choose a car solely by inspection from the outside or even with a test drive. I wouldn't be able to know whether a vehicle was safe, efficient, or reliable over a reasonable lifetime. Fortunately, I don't need to know determine these things for myself for two reasons - we have legally mandated safety and efficiency standards, and the workings of cars are not secret, so other people who know more about cars than I do can examine them and publish their findings.
I wouldn't have to understand the complete search algorithm on my own. Just as with encryption protocols which are also hard to understand, openness would allow a large number of people to participate in the scrutiny. Nobody argues that security protocols should be kept secret because it would be preposterous to try to understand them.
The fact that you suggest that if I don't like Google, my only option is to use Bing as if that isn't much of a choice undermines your argument that the market is working well here.
And no, no other search engine is transparent to my liking. That doesn't make transparency into an unreasonable request - quite the opposite.
Frankly I find your position somewhat strange. Markets work well when we place demands on them, because then suppliers know what to make. If people demand a more open search engine, then perhaps Google will provide one, or maybe a competitor will produce one.
Your argument seems to be that I should not publicly articulate my needs, but instead only choose from what is already on the market and not complain about the aspects I find inadequate. I don't understand what good that's supposed to do.
Agreed.
I wanted to make the point that computers are providing more direct feedback into our lives beyond simulators and data stores. I did use stronger language on purpose. It is a bit of a leap to say a computer told you what to do, but it is not that far from the status quo of suggestions.
Facebook example was a bit more of a stretch. I don't think dating sites have very sophisticated automatic algorithms, but something like a game opponent matching system might be a better example of an autonomous system that directly results in a social interaction.
Google Maps tells me the whole route, not just 'what to do next'. I can understand how the route was produced, and because it's shown on a map, I can see whether it matches my intentions before I follow it.
Amazon is less transparent, but they usually do show things like 'we recommended this because you bought that, and 40% of people who bought that liked this"
I'd hardly say that Facebook tells me who to I should be friends with - they mostly suggest people I might already be friends with to make it easier to connect with them. I don't know what algorithm they use, but I guess it's based on mutual friends and their scrape of my email box from ages ago. I wish they would tell me more.
Google on the other hand is completely secretive about how they generate their recommendations (i.e. the search results). How do I know the results I see are aligned with my interests?
Since Google's customers are advertisers, it wouldn't be unreasonable to guess that the search results are at least to some extent tuned to maximize advertising revenue for Google, as opposed to my needs for quality, breadth of coverage, historically significant results etc.
The algorithm may not be tuned to favor specific interests, but it clearly has a structural bias that favors certain kinds of information over other kinds.
I think that having so much of the web's information flow governed by secret algorithms is very much a cause for concern.