
Microsoft attributes $80-$100 million dollars in revenue to a shade of blue - tbgvi
http://www.lukew.com/ff/entry.asp?1025
======
patio11
This is the frightening reality of what happens when you have a large business
where everything scales multiplicatively and then you make a trivial change
which is net-beneficial. People _want_ this to be impossible because it
strikes you as intuitively wrong that five minutes of work could be worth
eight figures. Reality doesn't care what you think, though.

Relatedly, and more prosaicly, if you average one successful A/B test with a
5% lift per month, and you do _nothing else_ , your business will grow at
about 80% per year. Multiplicatively effective businesses are a wonderful
place to be in life.

Then I look over at Zynga at all where you can do multiplicative effectiveness
within the context of a viral loop, which is just so effective I'm almost
certain it goes against my religion somehow.

~~~
fierarul
Don't take this the wrong way but on every comment I've read from you, there
is "A/B" :-)

Not sure how Zynga's effectivness might be against your religion, but "A/B" is
part of you somehow: religiously or not.

~~~
MaysonL
A/B = the core of the scientific method.

~~~
fierarul
Not so sure about that...

It looks to me that A/B leads quite easily to a local optimum.

But it's better than no method at all.

------
jefarmstrong
So the article says that #0044CC is the $80 million color. But if you check
bing.com they're using #0033CC. So what's up? Maybe it's a trick to get
everyone to switch to the obviously inferior #0044CC color while bing laughs
all the way to the bank with the real killer color!!

Bing - #0033CC Google - #2200CC DuckDuckGo - #3068DD Facebook - #3B5998 (what
where THEY thinking? ;-)

~~~
teamonkey
The point isn't that there's a magic colour, it's that they A/B split tested
the new colour against the old one and saw an increase in the number of people
clicking. It doesn't mean the process stops right there.

~~~
jefarmstrong
Yes, I was being sarcastic. :-)

~~~
teamonkey
I know, but I was being pedantic :)

~~~
OmarIsmail
and I'm being tautalogical :)

------
tbgvi
These are notes from a presentation by Paul Ray. It's an hour long so I
thought these would be better to digest. The $80 million color blue comment is
down by "Data Driven Process".

If you're interested in the full video presentation you can check it out here
<http://live.visitmix.com/MIX10/Sessions/CL06>

------
mtarnovan
"A specific color of blue (#0044CC) drove $80-$100 million dollars a year
increase over the light blue the design team tried first."

I call bullshit on this statement. There is no way they could have measured
this reliably unless the only thing they changed for an entire year was the
color of the links.

~~~
nandemo
Why would they have to change it for an entire year? Maybe it's enough to test
it for 1 day, or 10 minutes for all I know. If it improved clicking rate and
revenue by 9%(+-1%) and after 1 year they've got $1 billion of revenue, then
you could say that 8~10% of that billion likely came from Magic Blue.

~~~
mtarnovan
You want to sample for 10 minutes or 1 day and extrapolate for 1 year ? The
results would be meaningless.

Even if you sample 1 _continuous_ month you could have badly flawed results
(for example you might have sampled only early adopters, which might behave
differently; and that's just one example of the many, many factors that can
make signal indistinguishable from noise). Now I'm sure that the guys working
on Bing A/B testing know what they are doing, but I just can't believe a
change in link color would bring a "$80-$100 million dollars a year increase"
(from a total of ~2000 million $/year revenue which I extrapolate from this
[http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.m...](http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.mspx#income)
but which includes all revenue of the Microsoft Online Service Business - I
don't know how much of that is generated by Bing)

------
phoboslab
Huh? The article makes no sense at all. I mean WTF?:

"But something was troubling me with this number. In color theory the thing
doesn’t work like that. In order something to be complete red + green + blue
should be 255. But in this case we have (0 Red + 68 green +204blue = 272) And
272 <> 255."

~~~
radley
Each color goes from 0 to 255. It's not a sum.

------
jafl5272
Oh, good. They found the hyper-intelligent shade of blue from Hitchhiker's
Guide to the Galaxy.

(Actually, Google does the same kind of color testing.)

------
axod
>> "Guess what? Microsoft also tested multiple versions of blue for links in
their search results. A specific color of blue (#0044CC) drove $80-$100
million dollars a year increase over the light blue the design team tried
first."

You can't change a color, see things increase, and claim it's all down to a
change of color. Also as bing "increased", Live "decreased". Has Bing been
successful?

It makes for a fun PR story, but it's complete BS.

Whatever happened to science? How much money did Live search make? How much is
Bing making? Have they been able to add any traffic?

Some hard data would be interesting.

One point to note is that they essentially copied the colors straight from
Google.

~~~
lmkg
It's called A/B testing. You serve people one of two versions of a page at
random, and test which performs better. With enough data, you can be pretty
damn certain that the single change caused the results. It is, in fact, hard
science.

There are obviously a couple of caveats. First, you need to run the two tests
in parallel as you allude to, and second the separation between the samples
needs to be completely random. I would assume that Microsoft can keep these
caveats in mind.

~~~
unblogworthy
It think they'll need to do some A/B/A testing to be sure.

Switch back to the original shade only for those that experienced the new
shade. Maybe they'll see another multi-million dollar increase? That way you
could further prove that it's indeed the color that made the difference and
not simply the fact that the page was changed at all.

Added:

 _The Hawthorne effect is a form of reactivity whereby subjects improve an
aspect of their behavior being experimentally measured simply in response to
the fact that they are being studied, not in response to any particular
experimental manipulation._

<http://en.wikipedia.org/wiki/Hawthorne_effect>

~~~
bemmu
Personally I like to A/B test with new users. But I guess on an open website
like Bing (instead of one with logins) you won't be able to really know if
someone has visited it before.

------
Jim72
Interesting that the url of the search result is also green, like Google. I
just opened the two sites side-by-side and their results use mostly the same
color scheme:

Blue Title Black description Green URL

I would definitely say MS went with a the "familiarity factor."

------
vais
In case anyone's interested in pedantic details, the color appears to be
#0033CC (not #0044CC) and it can best be described as STRONG VIOLET (according
to the Color Curious iPhone app).

------
samd
So they found the missing shade of blue!

------
motters
Think I'll stick with DuckDuckGo.

------
dfranke
They should name that color "Hooloovoo".

