
Facebook announces next-generation Open Rack frame - el_duderino
https://code.fb.com/data-center-engineering/open-rack/
======
mlthoughts2018
This strikes me as a gigantic coordinated form of commoditize your complement.

~~~
walrus01
DC power wise, the original reason why they did this is that it's grossly
inefficient to have a 42RU worth of 1U servers, each with its own individual
110-240VAC to 12/5VDC power supply in it.

Or the equivalent of that with four-servers-in-2RU type setups, but also with
AC to DC power supplies.

They centralized the AC to DC conversion in a single unit in the rack, feeding
either 277VAC or 480VAC to each rack, and ran 12VDC to each server. The new
system wisely moves from 12VDC to 48VDC (same as most telecom equipment), and
probably has basic DC-DC converters in each server unit for 48 to 12VDC
conversion.

They've also gone with custom motherboards that entirely eliminate the 5VDC
rails which are distributed by 'normal' ATX server power supplies.

~~~
baybal2
There are 90%+ efficient reasonably cheap flex psus, but the mainstream is
around 80%.

So, you would not be saving much if you only have access to 110 or 220 mains.

The economy changes though if you can get access to industrial 3 phase linkup
at 400V, and can avoid additional low step down transformers.

~~~
vanderZwan
> _So, you would not be saving much if you only have access to 110 or 220
> mains._

"Much" is a relative term here, no? Like, doesn't that kind of depend at the
scale at which Facebook operates?

------
acd
Could ocp make a rack that behaves like a blade server?

Construct a server rack that works like a network Patch panel and server power
bus? Thus one would just install the server in the rack slot, no more Cables
to the server. Bonus award if a robot could slide in the server in the rack.

~~~
akhilcacharya
What’s the current industry practice for blades? Does AWS GCP, Azure use them?

~~~
Twirrim
> Does AWS GCP, Azure use them?

No. Cost / value is not in alignment there. You want your servers to be cheap
and disposable.

Plus lose the control plane and you lose all the blades within it. There's
just no value to it unless you're doing workloads that benefit from a high
degree of locality. Cloud services focus on having their networks be as fast
as possible so as to reduce the disadvantage of not being highly local.

Oracle (my employer), and AWS have been announcing various HPC cloud products
where we're starting to focus on highly local, servers with fast
interconnects, and it's _still_ not looking at blades. HPC workloads are
rather untapped by clouds so far, and it's a big market.

~~~
akhilcacharya
I work AWS adjacent and I wish there was more externally facing information
about how the DC's are operated. It seems like a really neat space.

~~~
Twirrim
It's more boring than you'd probably expect. Boring is simple, reliable and
cheap. The less opportunities for things to go wrong, the better.

------
baybal2
I wonder, when will facebook's hardware people start using their own
inventions?

I see them and other dotcoms pouring inordinate amount of money designing own
hardware without actually manufacturing or using anything of it.

As I know, facebook still buys very plain OEM servers from Quanta

~~~
electrum
Facebook does use it. They design their own hardware, but they are do not
manufacture it themselves, they aren't in the hardware manufacturing business.
They parter with companies like Quanta to do that.

Here's an older article that explains the relationship:
[https://venturebeat.com/2014/01/29/facebook-
quanta/](https://venturebeat.com/2014/01/29/facebook-quanta/)

~~~
baybal2
I'm well aware that. I'm saying that in a sense that they don't buy or use
much of the original "ocp platform" they show off at their events.

And much of it was said to have ended after limited deployment in their
Prineville DC, after which they switched back to regular OEM quanta gear with
just few things like blue-green handlebars added, and "barebone" motherboard
trays.

I've been whispered that the biggest buyers of the original "ocp platform"
gear is not even facebook these days, but some banks

~~~
nuclear_eclipse
Facebook's entire blob storage and data warehouse (multi-exabyte) is run
completely on OCP storage hardware built by ODMs, of which Quanta is included.
Anyone telling you that we don't use OCP is grossly misinformed.

Source: I was on the blob storage team when we migrated all of our data from
OCP's gen1 storage design [1] to the new gen2 storage design [2].

1: [https://www.opencompute.org/documents/facebook-open-vault-
st...](https://www.opencompute.org/documents/facebook-open-vault-storage-
hardware-v08-spec)

2: [https://www.opencompute.org/documents/facebook-bryce-
canyon-...](https://www.opencompute.org/documents/facebook-bryce-canyon-
storage-system-specification)

edit: corrected links.

~~~
baybal2
Well, then it is.

What do you use for regular servers these day?

~~~
nuclear_eclipse
More OCP hardware (or designs that will eventually be contributed to OCP), in
many different SKUs depending on use case. This page lists most of the
current-gen hardware that I am aware of:

[https://www.opencompute.org/contributions?refinementList%5Bc...](https://www.opencompute.org/contributions?refinementList%5Bcontributor%5D%5B0%5D=Facebook&refinementList%5Bfamily%5D%5B0%5D=OpenRack%20v2&refinementList%5Baccept_year%5D=&refinementList%5Bproject%5D%5B0%5D=Server&page=1&configure%5BfacetFilters%5D%5B0%5D=archived%3Afalse)

------
dana321
The sooner this centralized model of storing data ends, the better.

~~~
Twirrim
What does that have to do with an open specification for racks of computers?
This is a design that anyone could potentially adopt, and gain from the
engineering efforts that those involved in OCP have done.

There's even a marketplace you can purchase Open Rack components through (and
you could likely also go more direct to those companies rather than via OCP's
marketplace):
[https://www.opencompute.org/products](https://www.opencompute.org/products)

