
Challenging the Bing It On Challenge - msrpotus
http://freakonomics.com/2013/10/01/challenging-the-bing-it-on-challenge/
======
RyanZAG
Microsoft lying in an advert? Impossible! They've never had to resort to that
before!

[http://www.curi.us/1571-lying-microsoft-
advertising](http://www.curi.us/1571-lying-microsoft-advertising)

[http://blog.debate.org/2010/12/06/the-cloud-is-a-lie-how-
mic...](http://blog.debate.org/2010/12/06/the-cloud-is-a-lie-how-microsofts-
to-the-cloud-ads-confuse-and-mislead/)

[http://techrights.org/2010/01/09/guardian-microsoft-
podcast/](http://techrights.org/2010/01/09/guardian-microsoft-podcast/)

[http://appleinsider.com/articles/13/05/23/microsoft-
caught-l...](http://appleinsider.com/articles/13/05/23/microsoft-caught-lying-
about-tablet-size-in-comparison-to-apples-ipad)

[http://www.youtube.com/watch?v=Di-
zq2JD-3A](http://www.youtube.com/watch?v=Di-zq2JD-3A)

[http://modmyi.com/content/10764-microsoft-bashes-apple-
samsu...](http://modmyi.com/content/10764-microsoft-bashes-apple-samsung-new-
nokia-lumia-commercial.html)

[http://www.theverge.com/2013/2/13/3984700/microsoft-
negative...](http://www.theverge.com/2013/2/13/3984700/microsoft-negative-
scroogled-ads-sign-of-things-to-come)

Of course, Microsoft has an article about why telling lies in your advertising
is bad, but as usual, they fail to heed their own advice.

[http://www.microsoft.com/business/en-
us/resources/marketing/...](http://www.microsoft.com/business/en-
us/resources/marketing/advertising-branding/why-lying-in-your-marketing-isnt-
worth-it.aspx?fbid=rsoNuhiWNFI)

~~~
cwoods
>[http://modmyi.com/content/10764-microsoft-bashes-apple-
samsu...](http://modmyi.com/content/10764-microsoft-bashes-apple-samsung-new-
nokia-lumia-commercial.html)

I am lost. What's the alleged lie here?

Also, two of your links leads to the ad which wrongly compares the screen size
of diagonal length is bigger.

However, when Google wrongfully claims a Chromebox with a Celeron is running a
Intel Core(TM) Processor, no one cares or makes big blog post on HN about it.

[https://news.ycombinator.com/item?id=4046964](https://news.ycombinator.com/item?id=4046964)

~~~
RyanZAG
My bad, I was trying to link to the Lumia 920 advert where they apologized for
lying after the fact. Here is a better link:
[http://articles.timesofindia.indiatimes.com/2012-09-06/hardw...](http://articles.timesofindia.indiatimes.com/2012-09-06/hardware/33648559_1_lumia-
pureview-camera-issues-apology)

The advert was apparently done by a Microsoft affiliated marketing firm, so it
made sense (to me) to include it.

In regards to your second point, Google and others often make _mistakes_ in
their adverts. Microsoft creates deliberate lies and seeks to create confusion
and falsehood. Much like your post, to be honest. Have you considered a job on
Microsoft's marketing department? I believe they are _desperately_ seeking
talent.

~~~
jmspring
Affiliated marketing firm means MSFT made the claim or guilt by association?

The Lumia 920 camera issue was wholly the responsibility of Nokia, thus why
they issued the statement. It doesn't seem unusual that partners on a device
might share marketing firms, so this link as an example is a bit of a stretch.

------
tomp
The article is good, but the math sucks. Statistically, 1000 elements is a
large enough sample for binary studies, and is consistent with the number of
people that are usually asked about their voting preferences on most polls.
Also, the size of the sample does not depend on the size of the population you
want to sample, provided that the population is large enough.

Now, whether the sample was actually representative of the population in
question is another matter.

------
justinsb
I tried the BingItOn.com challenge: 5/5 for Google for me. But I saw that
Microsoft was trying to push me towards "popular" queries (what Miley Cyrus
said today about the government shutdown). So I think this is an issue of
demographics and sampling: for mass-market 'leisure' queries I could easily
see Bing beating Google if that's what Bing is focusing on.

Equally, using Mechanical Turk is probably a biased sample (although to their
credit, the paper does examine the demographics in detail); mechanical turk
workers are paid more the faster they can find results, i.e. they are top-
result driven, whereas I would guess leisure queries are more focused on
entertainment and finding something interesting in the first few results.

And (despite the fact that I went 100% Google), I figured I should probably be
using something more tech-focused, based on my queries, like DuckDuckGo. From
that point of view, I think this BingItOn challenge delivers on the goal of
getting people to re-examine their preconceived notions of which search engine
they should be using.

~~~
mattwallaert
That's a remarkably clear-headed analysis; thanks for that. We at Bing don't
always get the courtesy of people actually thinking deeply about what we do.
=]

(Note: I work for Bing.)

------
pyrocat
It's great that they did this study. I wish more people would publicly
challenge many of the outlandish claims in advertising. But it seems rather
hypocritical to knock Microsoft for a sample size of "nearly 1,000" when their
own study "obtained 1,008 Bing It On challenge responses from the MTurk
platform and narrowed our analysis to 985 respondents who submitted screen
shots for 4925 searches."

~~~
yid
Not in this case, no. They were trying to replicate Microsoft's results, which
entails using a similar sample size.

~~~
justinsb
I don't think that's quite right. Imagine if I flip a coin and it comes up
heads, so I say "coins always come up heads". You don't flip a coin once to
prove me wrong.

I think the right thing to do is to conduct a much bigger study, get a much
better estimate of the preference distribution, and then derive the %
probability of getting a result at least as favorable as the Bing result by
pure chance.

If you can show there's only e.g. a 1% chance that the Bing result arose
through random variation, then Bing definitely has some tough questions to
answer. (e.g. did they run 100 different studies, and just cherry-picked the
best result).

~~~
yid
Yes, that would have been a great way to do it as well. However, the most
interesting aspect of the study to me was that they used Microsoft's
methodology, comparable sample sizes, and examined the effect of changing
search terms from Microsoft's "recommended" list to ones that study
participants suggested themselves.

Incidentally, you don't need a giant sample to estimate the variance on the
Google/Bing ratio -- you can use any number of resampling techniques like
bootstrapping to get that estimate.

~~~
justinsb
Good point - particularly as it's a fairly simple universe of answers. I
wonder if we could get the raw data. But, I do think the Mechanical Turk vs
"shopping mall" selection-bias factor overwhelms everything else.

------
neutralobserver
Hmm, they used Mechanical Turk "workers" to this experiment, paying each
worker 40 cents per answer, which netted them 400 responses. They upped the
payout to $1 and hit 1000 soon enough. They mixed the two response sets in the
final analysis.

I'm _SURE_ that didn't skew the results at all.

They respond to this in the study, but it's going to take a lot to convince me
that Mechanical Turk is a respectable source of subjects for consumer behavior
research.

The real news here is that Bing and Google are virtually tied with respect to
search quality. But that's been my experience based on using DuckDuckGo (which
is basically a rebranded Bing afaik).

------
ck2
Website operators need to revolt against Microsoft and their bots.

We see nearly a thousand parallel connections from their bots, nearly
continuously.

For only 18% of search use, they seem to consume several times more resources
than Google.

And for some stupid reason their bing bot and msn bot do not share data but
crawl the same pages independently.

(and yes we use crawl-delay directive, they ignore it)

~~~
nbuggia
You should contact the Bing team to follow up with this:

[https://support.discoverbing.com/eform.aspx?productKey=bingw...](https://support.discoverbing.com/eform.aspx?productKey=bingwebmaster&ct=eformts&scrx=1)

I used to run the Bing Webmaster program, and I know they do support crawl-
delay. It could be someone masquerading as BingBot, or simply a bug. Ping me
if no one gets back to you and I can follow up.

------
dangero
I don't think even the Freakonomics followup is valid because there's a huge
difference between telling someone to run some searches and actual search
engine use.

Here's what I might type if someone asked me to try their search engine:
"Miley Cyrus". The funny thing is that I'm not sure if I would like Google's
"Miley Cyrus" results better than Bing's. I'm not sure I could tell the
difference.

Here's the kind of stuff I actually need to work well during my normal day:
"xcode 5 warning signature expiration does not match provisioning profile"

The second one isn't a real XCode warning, and that's kind of my point. I
can't generate compiler warnings from my head to test Bing. I have to see them
on my screen first spontaneously. This means if I'm running a Bing challenge I
can't possibly simulate the real world unless I'm given my own machine and a
few hours at my leisure to run into these types of things.

~~~
GFischer
I'm at work, and I wanted to try the challenge, but BingItOn redirects me to
Bing.

Maybe it's not available outside the U.S.?

Bing's results are way less relevant to me than Google's, but Google does have
the advantage of a lot more personal data, and a specific subdomain for my
country (.uy) which often returns more relevant queries (OTOH for programming
I often have to force the U.S. website).

That's the reason I don't use DuckDuckGo either, the results are vastly
inferior for my use case (even though I want to like it).

~~~
mattwallaert
You're correct, Bing It On is US only (although I believe there is also a
Chinese version). As for Uruguay, we're always working to improve our
international coverage - check back soon!

(Note: I work for Bing.)

~~~
GFischer
Thanks matt :)

Competition is a good thing, I really hope you can improve on Google's
results.

I do happily use a lot of other Microsoft products, so I know you'll give a
good fight :)

------
DanBC
> When I looked into the claim a bit more, I was slightly annoyed to learn
> that the “nearly 2:1” claim is based on a study of just 1,000 participants.
> To be sure, I’ve often published studies with similarly small datasets, but
> it’s a little cheeky for Microsoft to base what might be a multi-million
> dollar advertising campaign on what I’m guessing is a low six-figure study.

He's right, but a sample of 1,000 is rigorous compared to most studies used in
adverts. Give a product to 50 people and ask if they like it or not. That
turns into 97% of people recommend our product!!

Since I live in a country with heavily regulated advertising I'm surprised
that there isn't something to cover "sounding sciencey" or "sounding mathy".
Certainly if the math is wrong you can get it changed.

~~~
coldcode
Sample size is immaterial unless you clearly can see and evaluate the criteria
under which they were chosen. 8000 of 9000 dentists choose Brand X toothpaste
means nothing if they asked 100,000 people and found 8,000 willing to say they
liked Brand X. By itself the sample size that is indicated may have little to
do with the actual test.

~~~
VLM
I was going to suggest the same thing, that given about 1/5 of people use Bing
(why?) then you simply sample 5000 people, select the 1000 or so Bing users...

However, it turns out there is an easier way. Simply select your sample from
existing heavy Bing users. Oddly enough, they'll probably prefer Bing.

~~~
mattwallaert
And this is precisely why a 3rd party does our testing: so that these sort of
manipulations don't creep in.

(Note: I work for Bing.)

~~~
VLM
Well fair enough then, I'll take your word for it. As some marketing advice,
beyond a certain (very low) level of savvy, people know the only reason
statistics are used is because they're manipulated. So, don't use statistics
in your ad campaign unless you want everyone above a certain (very low) level
of savvy to reflexively believe they're faked, which isn't going to help your
product.

Hey did you know that 9 out of 10 marketing execs endorse everything VLM posts
on HN? Yeah thats exactly what I'm talking about. Doesn't matter if it true or
even if it matters, its just not a good selling strategy.

------
rlu
For what it's worth, I'm pretty sure that most of us/the HN crowd can't fairly
do the bingiton challenge.

At least for me, it's really quite easy to tell them apart just based on
subtleties. I.E. Not the quality of the content itself but just how its
presented. And if there are ever images involved, it gets even easier because
Google and Bing display them completely differently (and honestly I think Bing
does a more elegant/"nicer" job of it)

~~~
rlu
Here's me "blindly" picking Bing 5 times:
[http://i.imgur.com/NAuhhhV.png](http://i.imgur.com/NAuhhhV.png)

This is why I'm skeptical of anyone here saying "I got Bing 100%" or "I got
Google 100%". It's extremely easy for us to game.

~~~
mattwallaert
This is why we don't track the Bing It On challenge itself and instead do lab
studies; on the website, it is fairly easy to game if you're trying to. Though
every once in awhile, having seen it a million times, I'm still wrong about
which one I think is which, particularly since Google changed their results to
look more like ours.

(Note: I work for Bing.)

------
sailfast
While the results returned are of use, Bing It On removes any of the right-
hand side Google content pulled from Wikipedia and other databases which
differentiate Google results quite a bit more than the Bing results
(especially for quick returns on top hits). Selecting only a portion of a page
is not really a fair comparison. That said, the effort did get me to take a
closer look at the two and evaluate which one I preferred. I just didn't pick
their horse.

~~~
mattwallaert
It is getting increasingly hard to do an apples-to-apples comparison (social
sidebar, snapshot, etc.). The best we can do is try to narrow down to just the
web results, for the moment.

(Note: I work for Bing.)

------
qwerta
Marketing aside I find Bing results good. Google is kind of turning into
singularity and tends to return only google (or US) centered results (for
example youtube only videos). Bing seems to have better diversity.

~~~
eitland
The opposite is just as annoying if not more:

I search in English and google insists on translating my search terms to the
local language and displaying local search results.

Again: idea for a startup: -like google but 5 years ago.

~~~
qwerta
Delete all cookies, visit google.com and tell it you speak english.

------
Pxtl
Bing's abysmal performance on WP7 has seriously hurt the brand for me. If
you're going to bundle the search engine into a dedicated button on your
phone, you should probably avoid making it awful.

------
ableal
I was curious about the test, but, for me,
[http://bingiton.com/](http://bingiton.com/) goes straight to plain
[http://www.bing.com](http://www.bing.com) . Geolocation, I suppose. I'm in
Portugal.

~~~
galapago
Bing is working worse in terms of results outside the U.S. They don't care
about it.

~~~
mattwallaert
Actually, Bing is working better outside the US than it has previously and we
continue to make heavy investments in other markets. But you're correct, Bing
It On only exists in the US and China at the moment.

(Note: I work for Bing.)

------
phamilton
Also important is the fact that BingItOn reports Google results different from
what I get when I search the same term on Google.com. My guess is that my
results get tailored to my profile on Google.com and BingItOn.com is
untailored. The thing is, tailored results are very useful. When I search
Ruby, I get much better results in my own Google.com than either set on
BingItOn.com.

------
BIair
tl;dr if you take the Bing challenge and make Bing your default search, try
other variations on Bing as you would have on Google.

I tried the Bing challenge, and changed my default search to Bing almost a
year ago. According to Google I was performing 10k - 20k searches per month,
so Bing Rewards was an appealing incentive. Over a years time, those are some
decent rewards.

Bing results are very good. The notable exception I've noticed is for
technical, geeky searches that are probably most likely to occur with this
crowd.

The biggest problem is one of "branding" and confidence. At first I found
myself searching Bing, and if I didn't find the result I wanted, I'd switch to
Google. The results were often very similar. But instead of returning to Bing
to perform another search, I'd do it on Google. As a Google user, when I
didn't find the results I wanted, I'd try a new search... I didn't try Bing.
When I noticed this and changed my behavior, I became more satisfied with
Bing's results.

~~~
mattwallaert
Well noted! I actually wrote about this very psych effect on the Bing blog; if
Google doesn't give you a good result, its your fault. If Bing doesn't, its
Bing's fault. This is common in situations where we go in with a bias.

(Note: I work for Bing.)

------
mVChr
Also, how well can you tell which search engine you prefer based solely on the
results page itself without investigating how well the content of the
individual URLs suggested in the results page satisfy your query on a one by
one basis?

------
nostrademons
When I've had my friends take the Bing It On challenge and polled them on the
results, they usually prefer Google by about a 2:1 margin, pretty close to
what Freakonomics found.

It's great advertising for Google.

~~~
mattwallaert
Are your friends people like you? Which engine do you like? Is it possible
that maybe your friends aren't a representative sample of the average
searcher?

(Note: I work for Bing.)

~~~
nostrademons
I'm referring to my college friends. I work for Google, and my Googler friends
are understandably biased; I don't include them in this sample. But I also
went to a liberal arts college, and my friends there are fairly representative
of educated young people in non-technical fields, and the same 2:1 ratio in
favor of Google seemed to show up.

~~~
mattwallaert
I went to a liberal arts college (Swarthmore) and I have to say: there is no
way my friends are representative of the average American. _grins_ Only one in
four adult Americans has a college degree. Average income: 30K. Building
search that appeals just to my friends is not my, or Bing's, mission.

(Note: I work for Bing.)

------
51Cards
BingItOn.com down for certain users? Just tried to open it here and it just
bounces me to Bing. I have tried it in the past so something has changed or
they have narrowed the focus market.

------
hayksaakian
A study produced by a party with an interest in producing a particular result?
It's more likely thank you think.

------
RyanMcGreal
When I go to bingiton.com it redirects to bing.com so I can't do the
challenge.

~~~
mattwallaert
Are you within the US? Unfortunately, Bing It On is only available in the US
and China at the moment.

(Note: I work for Bing.)

~~~
RyanMcGreal
Ah, that explains it - I'm outside the USA. Thanks for following up.

------
bifrost
Its a little interesting, I tried Bing for a bit to avoid Google's onerous
privacy issues, and I gave up. I moved to DuckDuckGo, haven't been back. The
results are usually more relevant for me and I don't have to feed the Big-G
machine.

~~~
mkr-hn
What privacy issues does Google have that Bing doesn't?

~~~
bifrost
There's a basically an open laundry list; its fairly extensive in terms of
tracking patterns/usage to violating their own "don't be evil" policy. Bing
doesn't have nearly as much paid placement, but that could be economic vs
policy.

Like I said though, I don't use either :)

~~~
mkr-hn
That didn't answer my question.

------
samspenc
Did you guys know...

BING = But Its Not Google

