
 One big cluster: How CloudFlare launched 10 data centers in 30 days - jgrahamc
http://arstechnica.com/information-technology/2012/10/one-big-cluster-how-cloudflare-launched-10-data-centers-in-30-days/?
======
ChuckMcM
This was an interesting article. Mostly because of how it took words that mean
one thing to people and then tries to define them as something else.

When we set up our "data center" in Santa Clara (nearly 1600 machines in 100
cabinets) racking and stacking and booting didn't take long at all, what took
forever was getting various network providers plumbed.

Co-location facilities range from extremely 'swank' like the Switch [1]
facility in Las Vegas, to the extremely 'bare' like some mid-peninsula
facilities I looked at. They can be run by a 'team' of folks with on site
security, network staff, mechanical teams, and HVAC teams, or one contract
security guy in a bullet proof glass booth noting who goes in and out.

A 'swank' co-location facility can sell you floor space at $800/kw per month.
A 'bargain' co-location facility might be as little as $150/kw per month. A
'cabinet' is generally between 4 - 10kW so $3,200 - $8,000 per month per
cabinet high end, and as little as $600 - $1500 per month per cabinet at the
bargain side. If you negotiate your IP transit (networking) costs you pay
between $3 and $6 per mbit or $3000 to $6000 for a gigabit line. For swank co-
lo centers like the ones mentioned you're also going to pay $150/hr for "smart
hands" services where someone on staff in the colo will go out and swap a
drive for you or replace a bad Ethernet cable.

The ownership of these places ranges from real estate investment trusts
(REITs) to businesses dedicated to internet services.

It is an amazingly diverse set of things for something as simple as chilled
room with power and security and network connectivity. The first time I had to
go up and do some work in Google's Oregon data center it became really clear
to me _why_ someone like Google would build their own data centers. The amount
of costs you could shave off your computing budget were _enormous_. With that
much profit just laying there on the table it becomes clear how folks can
compete in that space.

With regards to the article, if I want to drop 3 or 4 cabinets of 'stuff' into
50 data centers around the world using the local datacenter IP connection its
pretty straight forward to do. If you need a megawatt of space (I know
megawatts aren't normally a measure of volume but in Data centers they are)
and guaranteed redundant gigabit IP that can burst to 10Gbits, that is a bit
harder to co-ordinate in 30 day bursts unless you do most of the work
yourself.

[1] <http://www.switchnap.com/>

~~~
dsl
I think it would be more accurate to say "the install took less than 30 days."
If you get all your circuits dropped in and then start counting days, it
becomes much more realistic.

------
onetwothreefour
A bunch of machines in someone elses DC isn't launching a data center...

This article basically amounted to "we set up machines with some provisioning
tools".

------
iSloth
Looks like their trying to launch some new staff as well:
<https://www.cloudflare.com/join-our-team>

------
macros
Having done lots of this type of work I see it more as a triumph of paperwork
and scheduling skills than techincal acumen.

------
ck2
It's nice to see in the age of the cloud that co-lo can be "cool" again.

Ironically used by a so-called "cloud" company.

------
walshemj
well I wouldn't call a few racks in some one elses co-lo a data center.

~~~
robertfw
From the article, it appears that they are the sole resident in the data
center; "[Cloudflare] owned every router and every server in their racks".

~~~
biot
The word "their" refers to Cloudflare's racks, not the datacenter's racks.
Cloudflare might have two racks out of a thousand, but all the equipment in
those two racks were supplied by Cloudflare, not rented/leased from the
datacenter.

------
mnutt
They mention that they use anycast rather than geoip-located DNS. (which
itself would probably use anycast) I was under the impression that with
anycast it was possible for routes to change mid-connection and you could end
up with packets being routed to different datacenters as a result. Is there
something they do to prevent this, or is it not a problem in practice?

~~~
davidu
That doesn't really happen in reality. Convergence is fast and as long as your
datacenters are topologically disparate enough, you won't have random route
selection changes outside of failures, in which case, the connection would
have failed anyways. :-)

And if you're really tricky, you can sync state between POPs and keep a
connection moving if your application is aware of the state change. :-)

~~~
eastdakota
Yup. David knows what he's talking about.

------
devicenull
10 datacenters in 30 days really isn't anything terribly amazing. We're doing
about double that in half the time with about half the staff ;)

