
Facebook open sources its servers and data centers - arithmetic
http://gigaom.com/cloud/facebook-open-sources-its-servers-and-data-centers/ 
======
codex
This is a strategic attack on Google. A proliferation of scalable data centers
hurts Google a lot more than Facebook by enabling Google's competitors. Cheap
computation matters much more to search engines than social networking sites.

~~~
neilk
Because Google could never benefit from Facebook's designs too? Google does a
lot of things bespoke, but now they have a model to compare with that is using
more off-the-shelf tech. That can only help them in the buy-vs-build decision
process.

I think you're going overboard in thinking that everything the big players do
is about killing the other guy. Sometimes they just want to reduce costs. If
you decide that part of your infrastructure isn't strategic to own, it always
makes sense to be open.

~~~
brownleej
I'd say that efficient data servers are one of google's core competencies.
Google does not benefit from having that commoditized.

~~~
neilk
I just don't like the phrasing. "We did a thing that Google does not
substantially benefit from" == "We attacked Google" ?

------
flyt
Get all the CAD files and other specifications here: <http://opencompute.org>

~~~
swah
Any idea on how much would it cost to produce a small quantity of those
boards? At which scale does it starts to make sense to use a custom made
motherboard instead of a off-the-shelf PC?

~~~
flyt
HP, Dell and other vendors are already showing off hardware built to the new
specifications: <http://instagr.am/p/C66FU/>

~~~
hollerith
Your link does not support your assertion. It's just a picture without a
caption.

~~~
jedsmith
With a source fairly prominently displayed at the bottom. Follow along if you
absolutely must know more:

    
    
        1. Navigate to twitter.com/scobleizer.
        2. Observe first tweet.
    

<http://twitter.com/Scobleizer/status/56062315665178624>

That said, I lost all interest in the image due to the dumb focus effect.

~~~
smoody
ditto on the focus effect. reduces the value of the photo.

------
beagledude
say what you want about Facebook but I give them props for open sourcing so
much code to the community. Cassandra, Thrift, Scribe,Hive, etc...

~~~
lrcg
"Please take a look, tell us what we did wrong and join us in working together
to make every data center more efficient."

Mad Props.

~~~
skeletonjelly
Indeed. This is how I want to see society. Sharing knowledge for the greater
good. Probably the reason I'm not a fan of detriments of this such as software
patents and Apple in general.

~~~
farlington
Apple has open sourced a lot of their projects. Check it out:
<http://opensource.apple.com/> <http://developer.apple.com/opensource/>

------
whakojacko
Link to an article in James Hamilton's blog with actual pdfs of various
designs:
[http://perspectives.mvdirona.com/2011/04/07/OpenComputeProje...](http://perspectives.mvdirona.com/2011/04/07/OpenComputeProject.aspx)

~~~
wmf
As much as I like James and his commentary, I have this thing about primary
sources: <http://opencompute.org/>

------
corin_

      Sorry ARM.
    

Have ARM actually done more than announce that they will be moving into
servers? If they have then I missed the announcement. Either way, seems like a
fairly stupid dig at ARM, can't really expect companies like Facebook to have
moved onto ARM servers this quickly, even if it is the direction they intend
to go in.

~~~
wmf
This "sorry ARM" stuff is mostly just the media's need to turn every story
into a horse race. But there is some technical detail here: Facebook said
they're not ever willing to use 32-bit and ARM won't have 64-bit for years.
Also notice that Facebook's servers are "conventional" 2-socket with lots of
DIMM slots, not 1-socket "microservers".

~~~
steevdave
Fwiw, Freescale has a 64bit ppc processor now... Either way though, until
you're able to just drop an ARM processor into a "commodity" motherboard, it
won't really see widespread use, aside from cell phones, and other mobile
devices. There is also the fact that ARM processors all seem to have the
graphics card embedded in the chip.

~~~
wtallis
Current ARM SoCs have all been built for embedded applications, which is why
they integrate things like GPUs and radios.

The next-generation high end ARM core, the Cortex A15, will be the first one
that's intended to be suitable for server use: they've added virtualization
support, PAE for up to 1TB of RAM, and cache coherency to their bus to support
SMP. The server-oriented implementations will be quad-core 2.5Ghz chips, of
which you would be able to put at least 4 on a motherboard like this. When
they start hitting the market next year, they'll probably all but kill the
market for Intel Atom-based servers.

------
michaelbuckbee
I'd be very interested in hearing from someone with more experience in running
their own hardware what portions (if any) of what Facebook has announced today
is applicable at the small scale of say having a couple of co-located racks in
some datacenter. Maybe the base server designs?

~~~
cft
I can give you an interesting data point: our power bill is only about $600/mo
(2.5KWh), our bandwidth bill is about $3,500/mo (1.2Gbps). So power is a
smaller consideration.

~~~
patrickgzill
If you are in the USA, you are overpaying for bandwidth by about $700/month or
so.

------
rbanffy
I don't get the AC PSUs on the servers (<http://opencompute.org/servers/>).
Any reason why those are being used when each 6-rack group has a UPS (and
batteries) connected to them? Going DC would get rid of the inverter on the
UPS out and the 200 or so PSUs on each server.

~~~
lrcg
The UPS doesn't need an inverter - the PSUs have a DC in.

~~~
rbanffy
But they are supposed to use the AC input when not running off the UPS. I
wonder why that would be a good idea.

------
iloveponies
One thing I've found interesting: The decision to have batteries not per
server like Google, or per data centre, but per group of racks.

~~~
Swannie
Pretty standard in telecoms. Never quite understood why Google went per
server.

~~~
iloveponies
Their claims were that the efficiency was that great, and that it's acceptable
to pay the extra few dollars in motherboard manufacture to build it in.

~~~
Swannie
I guess in their instance the lifetime of the server was never likely to be
eclipsed by that of the battery. In telco, 15-20yrs of operation is not
uncommon, so a single set of replaceable/easily service batteries is
important.

------
budwin
This is a great talent acquisition play.

------
tallanvor
I'm sure they've done a lot of tests, but running servers in an environment
with 65% humidity just doesn't sound good for them.

~~~
ceejayoz
> In August 2008, Intel conducted a 10-month study to assess the effectiveness
> of using only outside air to cool a data center. The temperature range was
> 64°F to 92°F. Humidity varied from 4% to over 90% and changed rapidly at
> times. No increase in server failure was observed.

[http://www.energystar.gov/index.cfm?c=power_mgt.datacenter_e...](http://www.energystar.gov/index.cfm?c=power_mgt.datacenter_efficiency_inlet_temp)

------
skorgu
What are the chances of these being commercially available anytime soon? Is a
single consumer (even if it's facebook-sized) enough to jumpstart a B2C supply
chain? It's hard to see this catching on with anyone not building green field
unless it can come in at least on-par with a Dell/HP/Supermicro quote.

------
lrcg
Ethernet-powered LED lighting :D

~~~
oasisbob
I found that interesting too, thought it was a joke at first.

The only details I was able to dig up are in the Data Center specs [1], and
they're pretty brief:

> Energy-efficient LED lighting is used throughout the data center interior. /
> Innovative power over Ethernet LED lighting system. / Each fixture has an
> occupancy sensor with manual override. / Programmable alerts via flashing
> LEDs.

I wondered what the justification for PoE lighting could possibly be, sounds
like all the lighting is also functional as instrumentation.

Anyone know more?

[1]
[http://opencompute.org/specs/Open_Compute_Project_Data_Cente...](http://opencompute.org/specs/Open_Compute_Project_Data_Center_v1.0.pdf)

~~~
chopsueyar
A couple of guesses. They probably have a lot more cat5/cat6 around than
3-conductor 12 gauge copper wire (not sure if it is cheaper by the foot).

Also, the network switches can output the PoE, so maybe it is easier to wire
into/from the racks or overhead than a seperate AC line with conduit.

Labor installation and material costs may be less.

Also, since it is DC, and not an AC lighting source, interference may be less
(just a guess).

And as you mentioned, the lighting as instrumentation.

~~~
bhousel
All true. Right now 14/3 Romex costs about 3x more than Cat5. Copper is
expensive!

LED lamps use DC power. The LED lamps that you can buy that screw into a
standard Edison socket contain electronics to convert the AC line voltage into
DC. This is inefficient -- generates heat and wastes power (though still not
nearly as much as a traditional incandescent bulb).

By using power over Ethernet for their lighting, the datacenter can use
cheaper, cooler, more efficient bulbs, and save a lot of money on the wiring
too.

I can definitely foresee a future where new construction includes wiring for
both line voltage AC and also low voltage DC. It could eliminate all the bulky
transformers scattered around a typical house and save energy and money.

------
jacques_chester
Ah, economies of scale, is there anything you can't improve?

------
yanw
Not sure how many startups built their own servers anymore, this event seems
like a response to the Greenpeace accusations of Facebook's environmental
responsibility or wherever.

Edit: I agree it's a good thing, it's just that hosting a press event rather
than just making the announcement through a blog post suggests other motives
as well.

~~~
nroman
This is true. However, the services that startups use to host their servers
(Amazon Web Services, Rackspace, etc.) could adopt these practices in new
datacenters because the specs are open. This could lower costs for the
startups that use these services, and be good for the environment.

Seems like everyone wins.

~~~
vl
Any large scale service already adopted such practices, may be not to the same
degree. All large players buy custom hardware with specific tweaks to be more
energy (read cost) efficient.

------
ignifero
Shall we expect Google, Amazon, Microsoft et. al. to start posting their
energy efficiency stats from tomorrow?

~~~
wmf
Way ahead of you: [http://www.google.com/corporate/datacenter/efficiency-
measur...](http://www.google.com/corporate/datacenter/efficiency-
measurements.html)

~~~
Flenser
Looked through Google's numbers but they don't seem to be directly comparable
to Facebook's. Anyone know how far apart they are?

------
amitraman1
Wow.

------
davidpitkin
Facebooks attempts at openness make me wonder what they are trying to hide...
misdirection?

~~~
nanexcool
Are you serious? Facebook open sources so much of their code.

Check <http://developers.facebook.com/opensource/>

