

Bing smacks Google in new usability test - zuzzo
http://thenextweb.com/microsoft/2011/02/11/bing-smacks-google-in-new-usability-test/

======
brockf
Potential confounds:

* Google submits many more searches with it's instant AJAX search-as-you-type

* Google's speed allows users to hone in on better search results by tweaking terms, whereas Bing/Yahoo users don't waste their time with a 2nd search

* Google displays the answer right on the results page with superior semantic analysis, nullifying the use of a clickthrough

* Google's tools (such as "define:", math like "4+1.078", and conversions like "45cm in inches") are well-known and people look to Google itself for answers, not secondary sites in results

* If Google's users are more "tech savvy", they may use search qualitatively differently than Bing users (related to the above explanations)

* Google users click on Google Ads more than Bing users click on Bing ads - are ad clicks tracked?

Of course, these could all be wrong. But worth looking into or acknowledging
in the research if they already had.

~~~
corin_
Another one based on the logic that Google's users are more "tech savvy"
(which I believe they, on average, are):

* A larger percentage of Bing users are doing "time-saving" searches, such as "hotmail" and "facebook", where they know the first result is what they want even before they search.

And extending that logic - a tech savvy audience will be more likely to have
more complex results. It's not hard to give searchers want they want when
they're looking for "superbowl results" or "justin bieber".

~~~
gaius
_which I believe they, on average, are_

Self-selection bias?

Here's what I observe among my non-tech friends: they have Google as their
homepage. If they want Facebook, they type "Facebook" into the search box -
even tho' typing it into the address bar would do the same thing. If they come
across a PC that's not on Google, then they go to Google using the address bar
and proceed as before.

------
moultano
Matt's response to this:
[http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ...](http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ/This-
isnt-worth-a-full-blog-post-but-Ill-jot-a)

~~~
nostrademons
There's another possible source of selection bias: users may be going to
Google for queries that they _know_ are a hard, because they assume that Bing
won't have the answer. Of course such queries will fail more often, because
they're hard.

Bing's marketing has centered around the head of the query distribution -
common tasks where definite answers exist, like how to book a flight. It makes
sense that such tasks would succeed much more often than long-tail ones like
looking up an obscure error message.

~~~
barista
Well if they are "going" to Google for hard queries then wouldn't they try the
queries on Bing first?

Or are you saying that people spend time deciding a query is simple or hard
and then choose appropriate search engine for that?

~~~
nostrademons
More that they have certain problem domains where they might pick a given
search engine, and that those problem domains are biased toward using Bing for
easy searches and Google for hard ones.

I'll frequently search for airplane tickets on Bing, for example, while I
won't even go to Google for that. But when it comes to looking up obscure
programming stuff, I'm not going to bother searching on Bing, I know it'll
suck.

~~~
jamesbritt
"More that they have certain problem domains where they might pick a given
search engine, and that those problem domains are biased toward using Bing for
easy searches and Google for hard ones."

That makes no sense. Why would you, or anyone, alternate with search engines
if one of them too often fails to deliver? Why not just use Google all the
time and avoid the cognitive overhead?

Does Google do a poor job, or a worse than Bing job, with airplane tickets?

~~~
rue
I use DDG except for local sites, which it doesn't index anywhere near as well
as Unca Googs.

------
markkat
Here's an interesting comparison:

Google:
[http://www.google.com/search?hl=en&safe=active&q=Fan...](http://www.google.com/search?hl=en&safe=active&q=Fantasia&aq=f&aqi=g10&aql=&oq=)

Bing:
[http://www.bing.com/search?q=Fantasia&go=&form=QBRE&...](http://www.bing.com/search?q=Fantasia&go=&form=QBRE&qs=n&sk=&sc=8-8)

Yahoo:
[http://search.yahoo.com/search;_ylt=Agt64BoFz7.2ArgJZ_Io2BSb...](http://search.yahoo.com/search;_ylt=Agt64BoFz7.2ArgJZ_Io2BSbvZx4?p=fantasia&toggle=1&cop=mss&ei=UTF-8&fr=yfp-t-701)

What stood out to me is that images and videos are not being served up front
and center on Google.

No doubt more and more people are searching for video. Google really looks
old-fashioned in this match-up.

~~~
xyzzyb
My preference:

DuckDuckGo: <http://duckduckgo.com/?q=fantasia>

I prefer uncluttered and informative results at the top instead of news
headlines.

~~~
Legion
I do too, but I find for non-trivial searches, I often have to make use of !g.

More than once I've spent time doing multiple searches for the answer of a
tech question, re-wording the search to try and find a hit, only to have an
a-ha moment and say, "oh! hey! '(original query) !g'" and bam, first page, The
Answer.

------
baddox
I initially assumed that everyone around here would have tried Bing, then I
realized that may not be true. Have you guys given it a try? When Bing came
out I switched my browsers to Bing for a couple of weeks and noticed
essentially no difference in usability or results quality. Alas, after
reinstalling operating systems and browsers, Google gradually defaulted back
to the browser I use, and it's just not worth the effort to change again. I
really don't prefer one over the other.

~~~
arethuza
I don't use Bing for search (I'm tempted to start using DDG) but I do use
their maps quite often (though not exclusively).

I really don't have any brand loyalty to any of the search/map/whatever
providers - if something seems to be consistently better I'll switch (as I did
from AltaVista to Google).

------
aChrisSmith
Does my 'search' for "skitsophrenia" as a lazy-man's way of spellcheck
actually count as a query? IMHO this metric is flawed.

------
Uchikoma
50% of my Google use, as a non native English speaker, is for spell checking,
even for tricky native German stuff, USD to EUR conversion etc and I've also
never been using a calculator beside Google for some years now. Google just
does that fine.

------
kunjaan
"If you are like most people in tech, the last time you used Bing was never"

Woah. I am a SE and I use Bing's image and travel search all the time!

~~~
megablast
Are you saying you are most people in tech? He did not say everybody.

------
georgecmu
A study like this would only yield meaningful results if the user selection
for all three search engines were the same and randomly selected from the
general internet-enabled population.

In this case, one interpretation of these results is that google users look
for something very specific and thus are less likely to be satisfied with the
first result that pops up, while bing users are more casual and search for
generally consumable content.

~~~
brudgers
> _"bing users are more casual and search for generally consumable content."_

That would seem to be bad for Google. Bing gets to sell ads alongside casual
searches such as "cheap table lamps" and Google has to try to monetize
"'Company A' 143rd Ohio Brice's Crossroads."

------
suitcase
Wow, Google partisans need to calm down a bit.

The reduction in Big G's search engine monopoly is good for the market and
good for tech in general.

------
johnyzee
I am extremely disappointed with the new-fangledness in the newer Google
interface changes.

The 'instant' search is just a jarring user experience in the way it does
massive unpredictable updates to the screen at every keypress. Even worse is
the new image search results screen - mouseover on any image makes the search
field lose focus, it even does this when fetching the first page of results,
so you have to click the search field again every time you do a search.

Even the suggestion search field they introduced a few years ago was a dubious
value proposition - I personally got better value from the standard text
field's 'previous entries' than from the suggestions.

------
jjcm
I highly doubt their tactics here. They're grabbing aggregate ISP data for
users (<http://www.hitwise.com/us/about-us/how-we-do-it>) and running scripts
to see if the search is followed by an address that appears on the search
page. Also, it seems like they don't account for google's instant search, but
it's hard to tell as their first "success" statistic was posted in October
(Instant Search was released on September 8th). They do however, publish
upstream traffic percentages for sites in various categories
([http://www.hitwise.com/us/press-center/press-
releases/google...](http://www.hitwise.com/us/press-center/press-
releases/google-searches-apr-10/)) prior to September. Note that in this case,
the ratio between the upstream click throughs (75% for google, 15% for yahoo,
and 10% for bing for health related click throughs) and the ratios of the
aggregate search traffic (71.4% for google, 15% for yahoo, and 9.4% for bing
during that month) allow us to extrapolate the "success" rates during this
period, and it seems to suggest that everyone is in line for search results
(with google being slightly ahead, at least for health. I didn't run the
numbers for the other categories as we dont know the search volume for each of
those, and as such you couldn't find an average statistic that was relevant).
Just my two cents, but it seems like the data is highly affected by google's
instant search.

------
jeroen
"‘success rate’ .. a measurement of how often a user executes a search, and
then goes to a webpage from the set of links"

One simple example where this fails:
[http://www.google.com/search?hl=en&source=hp&q=circu...](http://www.google.com/search?hl=en&source=hp&q=circumference+of+the+earth)

The answer is shown three times in the results. There is no need to click on
anything. Did the testers compare these results for Google and Bing?

~~~
xentronium
[https://img.skitch.com/20110211-r3jmccgi1ejs24p1hbdnu69fwm.j...](https://img.skitch.com/20110211-r3jmccgi1ejs24p1hbdnu69fwm.jpg)

vs.

[https://img.skitch.com/20110211-cfdpqbca86fe1qgwdpbutdu153.j...](https://img.skitch.com/20110211-cfdpqbca86fe1qgwdpbutdu153.jpg)

Google's are better, however, they both don't show the value for me.

~~~
tuxychandru
I see it in the second result from Google!

------
electromagnetic
From my experience back in the pre-google days was that I'd have to open up
five or more links to ever see anything worthwhile and this was back in the
day when page titles had zero relevance too.

Plus, I haven't seen the google front page to search in months, so that's a
definite +10 for usability in my books. If I was smarter (or more pro-
procrastination) I'd have a link to google news on my chrome splash screen.

------
yakto
I call sampling error. Bing users are easily pleased with results, take
anything they're shown. Google users look harder, and often refine a search.

------
thesash
The explanation just _might_ be the fact that google submits new searches as
you type

------
kurige
I also find it interesting that a few defunct microsoft webpages (such as
<http://popfly.com>) redirect you directly to bing. Naturally, you'll follow
the wikipedia or equivalent link to figure out what's going on.

I doubt this would generate a large amount of click-through data, but it makes
you wonder what other tricks they have up their sleeves.

------
Havoc
>It implies that the user found what they wanted

No it doesn't. It will penalize a search engine where the user base like
tweaking the search & unduly boost the score of one with sheep for users that
click on the first thing they see.

------
kqueue
The number of users that use Bing is much lower than the number of users that
use Google.

This comparison is similar to saying that book X that has a total of 4
ratings, 5 star each is better than book Y that has 100 ratings, 4 star each.

~~~
jjcm
Given the volume of both though (they grabbed these stats from 10 million
searches) the statistical relevance of each should be pretty definite, even if
you're using a bayesian average.

------
bayshorecove
Google sucks in results recently.

------
gojomo
My greatest search success is when the information I need is in the displayed
snippets. Others clearly agree: see DuckDuckGo's 'zero click info' and
Google's 'One Box' results.

Next best is when the snippets make it clear I need to refine my search,
rather than wasting my time wading through the result sites. Again, no clicks
from the results, but a happier searcher.

So the Hitwise 'success rate' metric seems totally bogus to me – _inversely_
correlated to the quality of snippets/OneBox.

I suspect Google's snippets are still the best in the business, though an
objective analysis of that would be interesting news.

~~~
moultano
When powerset launched they had very impressive snippets. Now that they've
been bought by microsoft, I wonder how much of the technology has been
integrated.

------
mih
Noticed how Yahoo! has a higher success rate than Google? That alone is proof
that these numbers do not correlate to the real world experience.

------
rimantas
Must have been wrong shade of blue.

------
ashishb4u
i think they opened google.com because bing was not good enough to come back
to!

