Google's strength is not in searching for head or torso terms like "facebook" or "2013 calendar". It is in searching out tail terms; those obscure terms which might include a typo or two, and the phrasing may not be correct.
Unfortunately (and I don't have much time to explain this) one simply _can not_ judge a search engine's comparative performance using random users. And the Bing guys are falling into that trap.
EDIT: To expand more on what I wanted to write.
// DISCLAIMER: This is my personal view, and has nothing to do with my employer.
There are 3 large (fuzzy) classes of queries on search engines: head, torso and tail. The head queries are small in number but large in volume: queries like "Facebook", "Google", "Yahoo", etc. (yes, people do 'search' for these on search engines. Just a handful of such queries may make up 10% of the entire search engine traffic, which would be 100M queries per day (or so) on Google. It is depressing to see the numbers). In general, any search engine worth its salt will get the head queries right.
The torso is basically deciles 2-6 or so (there's no hard and fast definition). Here you'd have most musicians, actors, popular restaurants, etc. Bing and Google both would do a mostly adequate job here, for the first result. But: if you want to explore the concept a little, Bing goes astray and Google does not. For example, search for "idle hand tattoo" on http://bingiton.com/ . The first result is the same, to my local tattoo parlor. But the second result on Bing is from Mesa, AZ; while Google sticks with the right result and offers up other, related sites (Yelp, etc.). Google knows that "Hand" in the title is not the same as "Hands", and gives me not only the store, but also their Yelp, FB and Twitter pages; in other words: Google _understands_ your query better.
And finally, the tail. Here we have obscure error codes, weird technical terms, etc.; most of which are seen maybe once or twice a day. These constitute the bottom ~30% of the query stream. This is where Google really shines. It is very rare indeed to look for an obscure term and find that Bing does a better job than Google (I have yet to find an example).
So, as a user, I have a choice of 2 search engines; and one of them I know will do a better job on obscure terms. Guess which one will I pick?
When Bing claims that "most people prefer Bing in our user studies" (or something similar), their conclusion is flawed because the study itself is flawed. You can't sit a person down in front of a computer and ask him to evaluate your performance on tail queries! Where are those tail queries coming from? If they are provided by Bing ("here's an obscure term, search for it and tell us what you think"), then the user has no way to evaluate how relevant the results are, because, by definition, it's an obscure term! On the other hand, if you ask the user to come up with an obscure term, it will most likely be a term they are familiar with and they'll already know the answer, so they won't really hunt for information. So in my "idle hand tattoo" example, they'll see the top result and claim satisfaction; when, in real life, I would like to see the Yelp reviews, maps, etc. for the place.
If they (Bing) really want to compare how well they're doing, here's a suggestion: setup a search engine (like bingiton, but with only 1 pane of results, and neutral branding), and make it the default for a large pool of users (with their permission). Randomly, switch the backend to Bing or Google. Then, monitor the heck out of what the users do (all with their permission): how often do they click on the first result, how often do they re-formulate queries, the time to the first click, how often do they quickly come back to the results page from a bad click, etc. etc. etc.
And this is why I use Google. I rarely need to search for "facebook"; I do have a brain. When I'm searching, its more often than not for something I can describe but don't know the name of, and Google blows Bing out of the water on these. Try describing a movie to see what I'm talking about.
It doesn't surprise me at all that Google tends to handle the long tail questions better. --They're much larger and have had many more years to collect data, build synonym dictionaries, etc.
When I tried to come up with obscure queries, Google does better at the technical queries, and Bing is doing better at the non-technical queries. But the technical queries usually weren't that far off.
Yes, they're much larger, bot not much older. Microsoft was in search business years before Bing, and I also think they now have Yahoo!'s dataset too.
They're just better in search. I generally dislike Google (because I don't like their business model, which is advertising [I don't care for privacy much, just hate ads]), but nothing is as good as them in search, nor could be (for the foreseeable future).
Absolutely agreed. I was also 5-0 Google. Examples: "mdc date" "python math" Google is the clear winner here, as well as many other scenarios where more specific searches yield incredibly higher quality results.
I find the Google+ spam annoying too, but my pet peeve is the redirects Google put on search results so they can tell which ones you click on. I hate this, first because it feels invasive and second because it noticeably slows things down. Most of the time it's not that bad, but every now and then it adds a second, or 2 seconds, or 5. I got so mad I astonished myself by switching to Bing for a while. And indeed the Bing experience was much faster. I've switched back to Google for now, mostly out of habit. But I've gradually gone from being a big fan of theirs to a grudging captive. Something could jolt me into a different orbit.
That's what they used to do. This being Google, I'm sure they had a good reason for switching, especially to something so corrosive of user goodwill. I've often wondered what that reason is. Anybody know?
Other than the logo, and the bar at the top that says "Join Google+" (which of course, may be a slight annoyance to some people), what difference do you find between the old Maps/Places page for a restaurant and the new "Google+ Local" restaurant page that you're afraid of being confused into landing on? I just checked a local pizza joint while not signed in to Google and those two things are the only things that aren't reviews and information about the place (i.e. the only differences of substance between the new pages and the old Place pages).
Well if your only complaint is the design of the page (which is a fair complaint, a poor design can def. be annoying), then the whole "stop trying to confuse me into using Google+ when I just want to use Places" of your original comment is sort of a red herring, isn't it? Aside from having the name "Google+ Local", it's essentially equivalent to having the Places page undergo a visual redesign which you dislike.
Sure. I logged into gmail.com yesterday morning, and my inbox was missing. I still had everything that "skips the inbox", like mailing lists, my sent mail was all there, but everything that was in by inbox had vanished.
I didn't use two factor before, but am now. The password I was using was unique, though, and also fairly complex, so I find it unlikely somebody brute forced it.
I tried the link you're suggesting, as well as looking through the trash etc. What seemed very odd to me was that the messages which started in my inbox were missing from he trash, but things that were "skip the inbox" weren't.
Meaning anything I had "trashed" before yesterday was intact.
Again this seems to suggest that the account wasn't compromised, but that I fell victim to some sort of bug. It would have to be a very strange/determined hacker to go through and move my inbox to trash, then go to trash and only remove messages that started in the inbox.
Is it possible that you accidentally typed *a# into a GMail window? Those are the keyboard shortcuts for "select all" and "move to trash".
Fear of doing that sort of thing has led me to disable GMail keyboard shortcuts altogether on my account. (To do that, press ? in a GMail window and click the link on the second line of the overlay window.)
Yesterday was some very odd behavior, so I must have missed them in the morning.
1) I logged into gmail, and saw only one email that I had gotten that morning, everything else was gone. I looked in trash, and the things in it were from over a week ago [I've been out of town at Burning Man for the last week], so things hadn't just been moved to trash.
2) Did the "recover deleted messages" request via google, and a few hours got a response that some things should be restored to "All Mail", but still didn't see much.
Ok, so I found it amusing that they don't include Blekko, I mean we did make this into a fun game with our three card monte game with our /monte tag in search results :-) And I suppose it doesn't really serve their interests to 'lose' to a little guy either.
That said, its an interesting experiment. And at blekko.com we've been running for over a year .
What we've found is that there is a huge brand bias, which is to say that if you use some search engine as your primary search engine, you tend to think that is the best one regardless of whether its Blekko, Bing, or Google (or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index).
But 'quality' is also a very subjective thing as well. So if you search for highly SEO'd categories you will find the Blekko and Bing do better than Google mostly because there is a curated input (in Blekko's case it was from day one with it's from slashtags and in Bing's they started doing outsourced curation (putatively after seeing how effective it is in Blekko :-) in some topics with their 'editors' program ) At some point Google will realize what Bing and Blekko have which are that the 'indexing the web' problem became the 'filtering the web' problem when the signal to noise ratio started decreasing in about 2005, and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.
If the Bing guys are reading (and I know you are) why not open up your challenge to the new kid on the block, we don't just do Blekko vs Google in our monte results :-)
This has come up a couple of times and I'd like to address it.
So one of the things that hinders the adoption/growth of search engines is muscle memory. Sometimes people type into their browser's address bar the address of a search engine, and then search, but more often than not they just click into the search box and start typing. If all the results are 'good enough' people don't bother to check who their search provider is, much less try others. You can manually change your search on Internet Explorer away from Bing to something else, you could manually change your search in Chrome from Google to something else, and you could change the default in the drop down box in Firefox to something other than Google. But if you don't know what you are missing, well you don't do that do you?
Now various sites which have bills to pay and no direct revenue stream, like download sites, offer a service to folks like Blekko which says "We can run an offer to our customers, which if they take us up on it you'll pay us for that." Just like Google gets paid by an advertiser if you click through their advertisement to the site and buy something, these services make your offer for you and then charge you if someone takes you up on it.
So that is pretty standard stuff, the down side is that this space has some pretty scammy operators. We have debated quite a bit about this internally. Our goal has always been to introduce people to Blekko who may not have heard about it (see above :-). However, we've also discovered that sometimes that policing these guys is a pain. They get paid if you install our toolbar (which gives you quick access to our search engine). We insist that they make it an option and clearly spell out how to continue on with your download or what not without choosing to install it, and we insist that the vendor of this software make it straightforward and easy to revert the install if folks aren't happy with it. So that leads to situations where lots of people download software and say 'no thanks' on the offer. Sometimes that annoys the site because they aren't getting paid and they "change things" so that it always installs whether you want it or not. When we find out about that we cut those people off. And if the toolbar doesn't uninstall we cut those folks off too. We had one vendor who gave us a 'final' release to QA where it worked easily and the actual release didn't. That really irritated us. Sadly there isn't the equivalent of the better business bureau for these folks, we discover them, add them to our list of bad actors and move on. We've put a number of changes in place to be better at managing the process and it may be that it ends up there are just too many scumbags in this space to make it a credible channel. Time will tell on that, in the meantime if you ever spot a site telling you that you must install a Blekko powered toolbar in order to proceed (which is to say its not completely obvious how to opt out) please report it so firstname.lastname@example.org and we'll fix it.
That's noble and all, but you know what, it's still a toolbar. If I see it in an installer it's still going to put a bad taste in my mouth. You know as well as I do that 95% of those installs are not intentional, or are manipulated. The kinds of users that install toolbars are not computer savvy, and when the computer says "you should install this thing", they're the ones to say "accept".
"so I found it amusing that they don't include Blekko"
Just took a look at Blekko. It looks good, but you can't honestly believe the layman cares as much about it as he does about search engines coming out of MS or Google. Does the layman (MS' target audience here) even know about Blekko?
So the back story, and the source of my amusement, was that we've talked quite a bit to Bing. And in those discussions they got to see the positive benefit of some curation, and got to play the 3-card monte game for themselves (and lose a lot :-). And then they went out and did 'editors picks' and now they've done this 'bingiton' site. So its always nice to be driving the conversation around search, and amusing too.
So do I think the 'layman' cares? Yes and no. I think they care that they get good search results, I don't think they care where they come from. As pg points out building a replacement search engine is a Hard Problem(tm). You can make a competitive search engine with just 15 - 20 million dollars, but displacing the existing player is more nuanced.
This is something Bing knows all too well, they effectively created a clone of Google's search capability and they only pull in about 30% of the market. Blekko has been steadily adding share primarily by not being the same as Google, rather we've added specific features that were valued by different segments of the market (we're still the only search engine that lets you completely turn off ads as a user preference for example) but mostly folks just use what ever the search box provides.
Honestly I'm not even a layman; I use DuckDuckGo and Google and read HN (obvs), but I've never even heard of Blekko. Looks nice, but I completely agree: it's a little disingenuous (or perhaps tunnel-visioned) for him to believe that people really know about it.
DDG is a great product, we are one of their partners and we are providing some of their search results. They are also quite popular on HN I think in part because of the cool things they have been doing about creating interesting one-box kinds of things. But that said, they aren't a search "engine" they are a search "provider." That isn't a bad thing, in fact its a good thing because you can get Google results through DDG without having to get Google's "value added" stuff like G+ friends feeds or being "bubbled" as Gabe is great at pointing out. But the difference is that if the search engines stopped providing results, DDG would stop being able to return them until they built their own index. As would a number of other places like search.com or dogpile.com etc. The weird thing about search is that until Blekko came along and built a third, there were really only two web scale indexes being used, Bing's and Google's. Amazing is it not?
So I think DDG is pretty cool, love the ideas that Gabe comes up with, and am happy to partner with him to provide access to our index.
To rebut your statement that I was being disingenuous however, I suppose it depends on how measure or define 'well known.' From the perspective of various traffic reporting companies blekko.com is a bit more well known than duckduckgo.com which is not a perfect measure either but it does provide a different perspective.
No, your not. But that isn't a bad thing. From your link:
"we do not expect to be wholly independent from third-parties."
That is the technical difference, search engines do expect to be, if not wholly, then at least materially independent from third-parties, they are the third parties, search providers are all about the experience.
But lets be really clear, that terminology distinction really only matters when you get behind the search bar as far as the world is concerned we're all search engines, even if, like search.com, they don't index anything, and while DDG provides tailored indexes which support key aspects of your site's experience. The differentiation is the experience in that case not where you get your data. And I fully recognize that this distinction is not important for a large chunk of the Internet .
For a general audience, yes, we're all search engines.
For a technical audience, no, we're not. And the distinction is whether or not you have a generalized web index or not.
For organic search they present to the end user in a nearly identical way, caveat the 'experience' benefits of one over the other.
But for other things, like "tell me all the sites on the internet that copied this article verbatim." or "give me a rundown on link authority to all inlinks to this page" those kinds of things you need both a web index, and rights to use it like that. That's a pretty objective difference in capability.
So in technical company (and I consider HN to be technical) I try to be crisp about the terminology, in non-technical company I refer to all of these offerings as search engines because it is less confusing and frankly they don't care what its called, they type in words and get results.
 My father in law (part of the 99%) thinks he logs into "Google" to get to the internet because Chrome defaults to Google's home page when it starts up.
I first heard about Blekko through HN. I first learned of their API which I used to build most of unscatter.com on through HN too. The 3 card monte the original poster of this thread referred to, once again heard about it on HN. HN has also had links to articles about Blekko's investments from Yandex.
These were all front page. Over all I think Blekko has done a great job of being relevant and covered on HN.
>and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.
You are aware that Google hires thousands of search engine quality raters, all over the world right? But rather than directly curating, i suspect they just use these humans to train their algorithms. There is likely some inductive step going on, in selecting which algorithm (or 'sort' of the web) is selected, and then filtered by keywords.
Secondly, most search engines, actually do also use end-user curation. I have funky enough search queries, that I often get google redirects as search results: they want to know (and record) which link i pick. Also, because of all the google-adds-cookies, google-analytics-cookies, etc, they likely have 'session-bags' full of destination links and search queries that happen to correlate.
For me personally, neither search engine did very good in this test. But that's likely because neither result set is personalized in this test. And i don't know, if Bing does that now, but even if it does, for me to switch to another search engine, means to train another search engine in my contextual world. Hell, i even notice difference in result sets in Google based on my operating system. I get different results in Linux, than i do on my iPad.
People that get a lot of spam in search results, really ought to login, and stop clicking those links.
"we've focused on building a better search engine by concentrating on what we think are long-term value-adds -- having way more instant answers, way less spam, real privacy and a better overall search experience."
Now I'd probably word it 'building a better search experience' and remove the duplicate at the end but that's just word-smithing. One of reason we love you guys is that we share the same values in this regard. The more 'engine-like' you get away from 'experience-like' the more dubious Google and Bing get with sharing their index with you :-)
And, like you guys, we like to partner with folks who can provide the 'long tail' as it were.
"or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index."
Looks like you might be wrong about that.
DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our own crawler), crowd-sourced sites (in our own index), Yahoo! (through BOSS), embed.ly, WolframAlpha, EntireWeb, Bing, and Blekko.
As a Google shareholder, I really want to know whether Bing search results are better than Google. And I really, really want to know what the trend is over time.
However I don't think Microsoft's study is any good. They say, "In the test, participants were shown the main web search results pane of both Bing and Google for 10 search queries of their choice."
So it seems that the study participant had to think of 10 search queries, one after the other. But this is not how search works. In the real world users need to perform one query every so often, and their choice of search terms is based on an immediate need rather than what comes to mind.
A good Bing vs Google study would ask 1000 people to use a split-test search engine for their browser search box, then monitor the results over time. I've asked HN about this before and got nowhere:
So is there business opportunity here? I imagine that there must be thousands of Google shareholders willing to pay £100 per year for the results of an unbiased, ongoing study into who gives the best search results. In fact, why doesn't Google set up the study and provide such information to shareholders?
Now, all of my searches were technical; I plan to try again later with more general, "normal people" searches. Bing may do better there.
Also, good results from even google searches have become harder to come by in the last year or so. I have to work harder at my search terms. I don't know if that's Sturgeon's Law at work, or if there's room for a search engine to improve enough to become better than google.
Perhaps targeted search engines are the answer (i.e. engines's who have both their search and crawler algorithms tuned for a specific target audience) are the answer?
Of course it's going to be a draw for most searches (especially the suggested ones: "math games", "coupons", "time zones"...), they're generic and easy to find.
You can use any search engine you want to find those things. Shoot, even AltaVista was good at those kinds of searches, and that was 10 years ago.
The ones I care about—even if I'm not a programmer or technical user—are the difficult searches. What if I need to find a specific car part that only ever existed in the 1972 Ford Pinto? What if I want to find a plumber who specializes in pre-20th-century homes? What if, what if, what if.
I used searches like that and more in this comparison, and it was 100% Google. I could even tell easily why it was Google, and it was pretty clear which one it was. I still chose Google (despite being basically unblinded) because it was fundamentally better at these specific searches.
That's what bing needs to compare. But will they show it? Of course not.
Edit: I did 5 more searches all non-technical but specific and picked only based on result quality. Again 5/5 google. (for those curious, the searches were: "lego mindstorms", "Mars curiosity hi-res", "home developments in myrtle beach", "used bookstores in portland, or", "top baby names in 1982"—the last one was very cool because Google also provided links to the 5 surrounding years right below the 1982 result. It's the little things you know...)
I've noticed this as well. The closer my searches are to what (I imagine) a typical Google employee does every day (Linux/Python/Java/HTML/HTTP/CSS, English language, American locale, someone in their 20's/30's, etc.), the better it does. Google search can be incredibly good at taking a partial Linux error message, and leading directly to a solution.
The further my search is from this, the worse the results I get. I'd say this is a common theme throughout Google's software. Google Wave, for example, seemed to be aimed squarely at the "I'm a Google engineer" use case. (I assume not many Googlers ride the bus, because that's a weak point of their maps!) The Chrome commercials they're running right now are a great example: they look like something straight out of the old demoscene, that Google engineers might do in their spare time, not something that I could imagine a use for.
I got Bing with 2-0 with 4 draws. I used 1 technical keyword and 1 "normal people" keyword in spanish. It was really tied though, most results were different but pretty relevant, very hard for me to pick a winner.
Why? Because I tested it with fairly specific queries. A while ago I tried to switch to Bing because after testing it with a few general searches, it seemed just as good. However, when I switched and started searching for compiler errors, etc - I soon switched back because the results were just nowhere near as good.
Here's what I searched for:
bash: !": event not found error
gtk calendar tutorial
python import gtk error
seed funding in uk
Something to remember by the way is that Google has spent a lot of time and effort on their "bubble", that is, search results personalisation (http://dontbubble.us) and it is a big part of what makes their results so great. That influence won't be present in this test.
Maybe I was imagining it, but four out of the five I did I felt I knew which was Google and which was Bing - not from doing searches that I would recognise, but just from the extra non-search information / lay-out.
I did my best to still pick without bias to my current search engine (Google) but ended up picking them 5/5 and Bing 0 times.
Interesting. The privacy implications are disturbing if true. I'm neither an IE nor windows user but I suspect most users wouldn't opt-in if given that choice. Any refs to research/evidence about the topic would be appreciated.
"When Suggested Sites is turned on, the addresses of websites you visit are sent to Microsoft, together with standard computer information. ... Information associated with the web address, such as search terms or data you entered in forms might be included. For example, if you visited the Microsoft.com search website at http://search.microsoft.com and entered "Seattle" as the search term, the full address http://search.microsoft.com/results.aspx?q=Seattle&qsc0=... will be sent."
Most people have little idea that allowing a feature called "Suggested Sites" will result in their Google searches and clicks being sent to Microsoft, or that Microsoft will use clicks on Google search results in Bing's ranking.
MSFT also uses something called the Microsoft CEIP (Customer Experience Improvement Program), and I think that's either opt-out already or they're making it opt-out in Windows 8--it's built into the "Use Express Settings," I believe.
Again, I haven't looked at this very recently, but if you're using a recent version of Windows and IE, you're probably sending your searches and clicks to Microsoft unless you've been very careful about how you configured your computer.
Agreed about "most people have little idea...". I wish more companies were more honest and open about these things instead of burying it in the fine print. I wonder what would happen if you enable "Suggested Sites" but then install IE 9+ (since it appears to support the DNT flag) and select "Do Not Track". The pessimist in me suspects it will just turn off tracking for all non-msft experiments.
In any case, I think you've convinced me that my decision to use a non-windows OS and a non-IE browser was the correct one.
For the longest time, I didn't even realize that clicking a Google result sent the click back to Google (I just didn't think about it).
Then I right clicked one of the results one day and did "copy link" since I wanted to send it to a friend, and got something like this (with a bunch of other stuff in it too that I've yanked out cause I have no idea if any of it would be identifying info, but the end result is that this link no longer works):
I just hadn't thought about it before. It makes sense that they want and use that info, but it wasn't obvious at all that they were collecting it in that way, since the address shown in the status bar is the destination site, not the actual link target. I had just always assumed that the results were based on links and such around the web, and didn't think clicks would be factored in.
Ended up picking google 4-1. I'm somewhat of a bing fan in the sense that I find them good enough for most searches. So I was a bit surprised that the results ended up skewing in google's favor ... guess that kind of backfired for them :-P
I got a draw and then a 5-0 for Google. A couple of result for Bing were weird, like displaying "Seattle" when I searched for "a" (the letter), NSFW images when I was searching for a novel ("accelerando"). Google knowledge graph it's just too much ahead ("age of mcgyver") and it has better results on maps and evaluating expressions.
5-0 for Google. I tried mostly programming related queries as I find the differences most noticeable there. Especially when searching for specific APIs or objects, Bing tends to give you just the project homepage and some unrelated pages, while Google gives you the actual deep link.
DuckDuckGo is now a hundred times better than Google for those exact & specific queries. I was looking for some info on my truck engine, and Google sent me to sales pages for completely unrelated parts (and sometimes, entirely unrelated cars!). DDG sent me to deep in an old forum thread, where someone had posted a diagram of exactly what I was looking for.
I've seen this in shopping results -- I'm probably going to buy from Amazon, but I like to see the top N competitors' results along side the Amazon results, so that I can easily confirm that no one is more than a buck or two cheaper than Amazon and can then feel good about my purchase.