
Google Has Spent $21 Billion on Data Centers - ajaxguy
http://www.datacenterknowledge.com/archives/2013/09/17/google-has-spent-21-billion-on-data-centers/
======
alanlewis
Headline is wrong. 2 data points that the article is based on: 1) Google has
spent 21 billion dollars on Capital Expenditures since 2006 (presumably
gathered from SEC filings) and 2) "Google says the majority of its capital
investments are for IT infrastructure, including data enters, servers, and
networking equipment." (source unclear)

So, Google has spent at least 10.5 billion on "data centers" since 2006. There
are notable disclosed expenditures that are clearly not "data centers"
mentioned in the article, including 2 billion spent to purchase NYC office
space. Still a huge number, but off by many, many billions.

~~~
swashboon
Didn't they spend 2 billion buying the 111 Eighth Avenue office in Manhatten
at the end of 2010? I believe that is data center realestate.

~~~
rads
Why would they put a data center in Manhattan? Don't these things usually go
where the space is relatively cheap?

~~~
nasalgoat
Latency. Putting things far away from your customers means response time goes
up.

~~~
_mulder_
True, but populous though Manhattan is, it hardly warrants its own data
center. If latency was their prime concern they would build the data center in
the middle of the population density. For North East US, a location between NY
and Washington DC would make more sense.

It could be for transatlantic latency, but this would just be between the US
and European data centers anyways.

In short, no, I don't think latency would be a big deal here. Correct me if
I'm wrong.

~~~
nasalgoat
When you're Google, 10ms makes a difference. In my own personal experience, a
decrease of 100ms in my response time results in a 20% increase of traffic.

Latency matters that much.

~~~
_mulder_
Yes 10ms does matter, but building a data center in the middle of NY vs 20
miles outside is not going to create anywhere near 10ms latency, its not even
going to create 1ms, on the Google network at least. Your cable provider is a
far higher source of latency.

~~~
nasalgoat
Depends on where the Meet-Me centre of the city is - it might even be that
building.

------
cobookman
My question is if google spent 21 billion, why did microsoft spend > 15
billion on its datacenters. How much datacenter resources is each microsoft
customer using? It can't be from just office 365, Azure, and Bing. So is the
majority from just software updates/downloads?

~~~
drill_sarge
Technet, Windows Update, Xbox Live, Hotmail, Bing, Skydrive and so on.

~~~
InTheSwiss
I often wonder what the bandwidth for Windows Update (and MS Download Center
for OS patches) is. I would love to see some details on it if it has ever been
discussed by MS engineers?

~~~
sanxiyn
[http://technet.microsoft.com/en-
us/library/cc627316.aspx](http://technet.microsoft.com/en-
us/library/cc627316.aspx) gives 500 Gbps for Windows Update in 2008. (2008
announcement:
[http://blogs.technet.com/b/mscom/archive/2008/04/11/announci...](http://blogs.technet.com/b/mscom/archive/2008/04/11/announcing-
the-launch-of-the-microsoft-com-engineering-operations-techcenter.aspx)) I
can't find recent update.

~~~
InTheSwiss
Thanks!

------
jcampbell1
Does any one know the relative costs of hardware vs building and cooling? If I
had a cool billion to spend, how much would I spend on
building/cooling/power/generators vs servers and networking hardware?

~~~
ChuckMcM
Yes, and they aren't particularly relative :-)

But to answer your question you can get pretty close by translating into units
of watts and remembering a couple of simple rules.

Rule one is you only use 80% of the provisioned power (this is a safety rule
with respect to things like circuit breakers and infrastructure, it keeps
random surge events from destroying everything).

Rule two is that your power use efficiency (PUE) ratio determines how much
cooling you need.

So lets say your PUE was 1.3 (that means total power / computer power = 1.3,
the extra going into cooling) and you were building a 10 Megawatt data center.
Well you have 8 Megawatts of actual power (80%) and 8/1.3 or 6.15 Megawatts
for machines. Going by 750 watts per server (the PSU label [1]) you can put
6.15/.00075 = 8,200 servers into that data center. At 500 watts/machine you
are 12,300 machines. So to make the math easy lets say it is 10,000 machines.

If you servers cost you $5,000 each then you're at $50M in server hardware for
a 10MW install. Network switches are $100/port (assuming you're not doing 10G
ports) and cable is way more expensive than it should be adding another $20
per port to connect it to the switch, so another $1.2M there. You need to hold
them in something and rack solutions go from the less expensive (4 post) to
the more expensive (fully enclosed) with a range of $500 - $2500. How many you
can put in a rack will change your per machine rack cost up or down, but its
somewhere between $25 - $125 per machine so another $1.2M so lets be generous
and say our $55M for your 10,000 machines which are going to consume 8MW of
your 10MW provisioned power.

Building the data center though, especially if you include land acquisition,
that is going to be pricey (of course you can write it off over 27.5 years but
still) If we start with the fact that a data center is really just a big
warehouse with a cold box inside, we can look at building a warehouse [2] and
some folks have some priced out [3] which put it about $50 - $75M range, and
that is before you add cooling, back up generators, and pay the local utility
to run a 10MW power line into your building. I got some estimates from Crane
for a 'pocket' data center project I was looking at (about 3MW) and those
extra costs are probably going to exceed $100M (split 33/33/33 into cooling,
back up power, and secondary construction).

So back of the envelope 3:1 construction costs over equipment cost. And maybe
$20M per constructed MW of compute.

[1] This isn't what you would do since machines don't consume that much, they
can only not consume _more_ than that, but whatever.

[2]
[http://www.reedconstructiondata.com/rsmeans/models/warehouse...](http://www.reedconstructiondata.com/rsmeans/models/warehouse/)

[3]
[http://www.mohavedevelopment.org/media/docs/One%20Sheets/Moh...](http://www.mohavedevelopment.org/media/docs/One%20Sheets/Mohave-
County-DC-Construction-and-Amortization-Costs-2012.pdf)

~~~
JabavuAdams
Thanks for the informative post. Are you aware of any non-proprietary tools
for doing procedural generation of data-centers?

I'm imagining a generate-and-test cycle given estimates for the various
component prices.

~~~
ChuckMcM
I don't know of any off the shelf ones, but there are "data centers" like
Google and Microsoft and Amazon build for their "gear" which they resell as
"services", and there are "data centers" which are built by Real Estate
Investment Trusts which are designed for 'unknown' gear and 'unknown'
services.

It is more expensive to build the latter since you cannot know ahead of time
what gear to support. Whereas if you're building a box full of Open Compute
servers you're in a much better space.

The "services" data centers are really much closer to building sized servers
than they are hosting facilities for rack mount servers.

I expect like other specialized facilities part of the 'value add' as a
construction agent is both procedural generation tools and experience with
various cost trade offs which really only comes from building several of them.

------
antidaily
So they should use Hadoop?

~~~
wikwocket
I'm not sure if you're posting sarcastically, but Hadoop was built based on
whitepapers Google released about MapReduce and the Google File System. I
think they are covered in this department!

~~~
DennisAleynikov
They're posting in reference to [1] I believe. It's supposed to be a funny
comment. They are pretty damn well covered in this department, I agree.

[1]
[https://news.ycombinator.com/item?id=6398650](https://news.ycombinator.com/item?id=6398650)

------
AznHisoka
What if you were trying to start Google today (assuming it didn't exist), and
you had to bootstrap with maybe only $500 available to spent per month? How
would you guys do it?

~~~
kintamanimatt
I presume you're talking about hardware budget. You probably wouldn't get very
far with such a low budget, but I think your question is how to get the most
bang for your buck. The answer is with dedicated hardware, especially for
computationally expensive apps. $500/month dedicated hardware will beat the
pants off of $500/month of EC2 instances for example, but it comes with the
cost of being less flexible.

------
outside1234
is that all? that's only $10k per machine if they have 2M machines (which I'd
guess is a conservative estimate)

NOTE: I'm not being snippy - that seems very efficient given the building,
HVAC, networking, and the fact that they have probably had a lot of machines
fail.

------
znowi
How much of it was subsidized by NSA?

[http://www.theguardian.com/world/2013/aug/23/nsa-prism-
costs...](http://www.theguardian.com/world/2013/aug/23/nsa-prism-costs-tech-
companies-paid)

~~~
jonknee
Nearly nothing considering your article mentions "millions" for the whole
program and Google has spent 21k million just on its own. Google makes more on
interest from its cash on hand than it has received from the NSA.

~~~
melling
It's really disappointing that you're responding to what is the #1 comment.
The NSA noise just overshadows much of the meaningful discussion on HN.

~~~
DennisAleynikov
It is quite disappointing. I'm personally very tired of how widespread and
acceptable the conspiracy theories have become, everyone's wearing their
tinfoil hats and running around screaming like the government's taking away
their first born child. I can't say I care anymore. But that's like, my
opinion man.

------
ffrryuu
And I have spent tens of thousands on rent.

------
domdip
Has anyone dug in and figured out how these #s are impacted by MMI?

------
pearjuice
They do have 100% uptime, right?

~~~
socillion
Google had a 5 minute outage this August, where global internet traffic
dropped by 40%.

[http://www.fastcompany.com/3015942/most-innovative-
companies...](http://www.fastcompany.com/3015942/most-innovative-
companies-2013/a-brief-google-outage-made-total-internet-traffic-drop-by-40)

------
a3voices
I wonder how much money Google has spent per user. $10? $50?

~~~
panacea
...and did they improve the fortunes of their advertising client base, or
merely suck the wind out the prevailing sail without offering anything in
return?

~~~
lsc
It is a valid question. As a 'brand building' advertiser, I never felt like
most of the google products delivered as much value as they claimed; they all
seemed to be targeted at products people will buy after the first impression
(which is... not, generally, how products in my sector are sold. People buy
because they have heard of me. The 'hearing about me' is important, not the
last click.) and the tracking mechanisms all seemed... naive. (now, that last
part might just be the effect of them dumbing it down to the point where a
small business owner, like me, can understand whats going on, but man... in
that case, we advertisers trust google a lot.)

Of course, I'm small, cynical, stupid, and don't spend much effort on
advertising, so it's possible my impression is wrong.

------
tunisian123
google doesn't throw her money away !! I'm positively sure that they will get
it double a triple :) they don't do charity

~~~
dudus
Google actually does a lot of charity

[http://www.google.com/giving/](http://www.google.com/giving/)

~~~
praptak
Publicly traded companies even if they do "charity", they do it expecting good
PR in return. Any other kind of charity (i.e. done without publicity, only to
make the world a better place) would be against their duty to the
shareholders.

~~~
baddox
You're just bringing up the tired thought experiment about whether "true
altruism" exists. Is it really corporate "charity" if the company expects good
PR and eventually profit as a result? Is it really individual "charity" if the
person gets a really good feeling as a result?

