
The end-to-end refresh of our server hardware fleet - sloanesturz
https://code.facebook.com/posts/1241554625959357/the-end-to-end-refresh-of-our-server-hardware-fleet/
======
cortesoft
I really dislike this sort of naming scheme (Bryce Canyon, Honey Badger, Mono
Lake, etc)

The names tell you nothing. You can't tell which one came before which, or
even what they are. You just have to KNOW that information. A good naming
scheme tells you information about the thing named.

~~~
hkmurakami
Is there since information in Intel codenames I'm perhaps unaware of?

~~~
krylon
I haven't been able to keep those straight for years. Maybe this is just me
getting old, but I miss the old days, when you could easily tell that a
Pentium is faster than an 80486, and that a Pentium 133 is faster than a
Pentium 100.

These days, CPU speed matters less than it did back then, but there still are
CPU-hungry applications (I'm looking at you, Autodesk Inventor!), and if I had
to put together a PC from scratch (which I think I'll actually sometime this
year), I would be kind of lost.

~~~
foota
Part of the issue here, I think, is that cpu's are much more complex than they
were then. You have a number of different cpu lines with different models on
the market at any time.

~~~
krylon
That is true.

But that does make the decision what CPU is best for a given use case and
budget much more complex, too.

(Like I said, the impact of the CPU on overall system performance is less
today than twenty years ago for many use cases, so it is not _that_ much of a
problem.)

------
JohnJamesRambo
So much equipment and money devoted for something as pointless as Facebook. I
wish it could go for something cooler and more useful. Something that hasn't
been shown in studies to make us feel lonelier.

~~~
Swizec
As a small business advertiser: Facebook is awesome. Nowhere else can you get
$400 of sales for $10 in ads. That's a 40x ROI.

~~~
jonknee
Your ROI is from profit, not revenue. You can easily get $400 in sales for $10
in ads if you don't care about profit.

~~~
nickpsecurity
You can get it for $0 in ads on Craigslist. ;)

------
hyperbovine
The last time hey did this it flooded the market with dirt cheap E5-2560s. Are
we in for a new updated deal?

~~~
anderspitman
I'm writing this on a dual E5-2670 v1 system I built for about 1000USD (not
including monitors). 16C/32T, intell s2600cp motherboard, 48GB ECC, 256GB SSD
(some cheap one), dual 3TB WD Red. Running docker and kvm on the host, and
also currently using it as my main desktop. I have a freenas guest with ZFS
handling the HDDs (PCI passthrough of a SATA controller), and a Windows 10
guest with GPU passthrough for office and games. All-in-all it's been a fun
project, especially for the price. Incredible how cheap hardware can be these
days. I paid 60USD for the CPUs when I did my build. I think they've gone up
about 50% since then, but are still a steal in my opinion.

~~~
krylon
Just out of curiosity, do you have an idea how much power that system uses?

I will probably get myself a new desktop computer sometime this year, and
those specs sound pretty sweet. But I want to keep an eye on power
consumption, too, and I don't my desktop to either melt or have its fans
create a tornado in my living room... ;-)

~~~
anderspitman
I went ahead and rebooted it so I could plug my Kill-A-Watt back in just for
you ;). It idles at right around 110W. I did a quick sysbench to peg all the
cores and it goes up to about 300W. Note that I have some other things sipping
from the meter so it's actually several watts lower for just the tower. With
the Windows 10 guest running (host using GTX 950, windows RX 460) it idles
between 190 and 250.

~~~
anderspitman
BTW I couldn't really tell any difference in the fan noise levels when the
CPUs were pegged, though the temps went from ~40C to ~60C. It's already pretty
quiet. Just barely above the level where I notice it.

------
loser777
I hope some of the old hardware makes it to ebay, but it looks like many of
the form factors are proprietary.

~~~
jacquesm
What do you intend to do with it?

I've found that almost any kind of short lived experiment I can do cheaper on
AWS than doing it with hardware that I own. If it is longer running then it
might become viable to own the hardware.

~~~
hueving
It's sad to me this is becoming the status quo. Using other massively
centralized companies for compute resources is a sad future.

It's bad for privacy, it's bad for diversity to protect against SPOFs, it's
bad for general computing hardware (vendors primarily target the giants), it's
bad for users via vendor lock-in, and it's bad for open source projects in the
infrastructure space.

I think hackers justify it to themselves by pretending it's a commodity like
electricity, but it's far from that. If my utility goes out, I can turn on
generator and get exactly the same electricity. If Amazon goes out I have to
build again on another cloud from a (hopefully recent) backup or just sit dead
(like the recent s3 outage).

Sorry about the rant, but is there anything that would get you to stop giving
the keys to the kingdom to Amazon?

~~~
raz32dust
I feel you. But I think the lock-in problem can be solved if we can have some
standardization of cloud services such that you can always move to another
provider. That has to start with some big company developing an abstraction
layer and open sourcing it, and then we can go from there. I think Netflix has
a switch from Amazon to GCP; I hope be they'll standardize it and open source
it.

~~~
cowardlydragon
Chef, puppet, etc, and other Apache projects all offer this already.

It's like leveraging Oracle specific database features. Your a fool to do so.

------
nodesocket
How long until Facebook joins the public cloud business with Amazon, Google,
and Microsoft?

~~~
wmf
Never? Their infrastructure is cool but it's only around half of what a public
cloud would need.

~~~
nodesocket
Never... I'm not convinced. Rollback to when Amazon was pre AWS. Everybody
thought they were crazy announcing they were getting into the datacenter and
cloud business. I'd say it has worked out well for $AMZN.

~~~
dragonwriter
> Everybody thought they were crazy announcing they were getting into the
> datacenter and cloud business.

All the comments I heard were positive about how they were diversifyiny by
leveraging expertise they had been forced to develop for their own core
platform, not that they were crazy. I'm sure there were some.who.said "crazy",
but it definitely wasn't everyone.

~~~
addicted
I agree. Maybe Amazon sold it really well, but as far as I can remember, the
response almost universally (including in financial circles) was that this was
a great idea since it allowed them to leverage idle resources they needed to
build out to handle peak loads (such as during the holiday season).

~~~
canes123456
That never made sense. What happened during holiday season? Everyone on AWS
was put on hold?

~~~
gregmac
Amazon.com, even at its peak computing needs, is now a drop in the bucket.

Three years ago, in 2014, AWS was adding the equivalent hardware _every day_
of what ran Amazon.com in 2004, when it was only a $700-million company. [1]

[1] [https://www.enterprisetech.com/2014/11/14/rare-peek-
massive-...](https://www.enterprisetech.com/2014/11/14/rare-peek-massive-
scale-aws/)

------
dexterdog
100 million hours of video played per day. Are people actually watching this
video or are they just inflating the number?

~~~
Veratyr
They've inflated metrics in the past:
[http://www.businessinsider.com/facebook-video-views-
exaggera...](http://www.businessinsider.com/facebook-video-views-
exaggerated-2016-9)

I'm not sure if there's any standards between platforms for these things that
allow you to compare though. I'd say for example that you should exclude
watches that last less than 5s or so. YouTube and Netflix may not have thought
to do it because it doesn't make much sense to them but Facebook really needs
to since I assume most of their video watches are automatic (accidental) while
scrolling through the feed.

~~~
dexterdog
It does matter to Netflix. They don't publish their numbers and just use them
for internal metrics so you can bet that they are honest with themselves about
their numbers.

------
jabl
I'm disappointed in the "open rack" designs. For a really minor improvement in
density they have broken compatibility with standard 19" gear.

One could argue that at FB scale it's worth it, but then MS seems to manage
just fine with 19".

------
oso2k
It's interesting. If they wanted to, they could compete with the likes of HP,
Dell, Lenovo, and Cisco if they could ramp up production to accommodate
customers. I wonder who does their manufacturing on the backend.

~~~
wmf
Facebook uses Quanta/QCT, Celestica, and Accton for a lot of their
manufacturing. You can buy Facebook servers from companies like Hyve, AMAX,
and Stack Velocity but they aren't really aiming at the mainstream server
market.

------
quickben
Is it that cheaper to custom build, if you can't ebay them off at their half
life point to recover some of the cost?

~~~
wmf
Most of the cost is in processors and RAM so those parts can be sold at end of
life. There are server recycling companies that specialize in this.

~~~
saycheese
Here's an example of one:
[http://cashforelectronicscrapusa.com](http://cashforelectronicscrapusa.com)

~~~
PhantomGremlin
Yeah, that kinda reminds me of expertsexchange.com

Is it electronic-scrap or is it electronics-crap ?

~~~
saycheese
The terms of use page refers to "CJ Environmental" which when combine with the
INC 500 reference of the home page returns this profile:

[http://www.inc.com/profile/cj-environmental](http://www.inc.com/profile/cj-
environmental)

From the FAQs, "How does material get processed and refined? All materials are
unique and subject to different methods of processing. Newer computers are
refurbished and given new homes to maximize ROI for our customers. Scrap
product is crushed or shredded before the refining process. All material is
processed in accordance with all Federal, State and Local regulations. To
learn more about our licensing and compliance measures contact us."

------
taf2
When can we start hosting our services with Facebook - similar to aws, gce,
etc?

------
trustfundbaby
Anyone know what cpus/gpus they use in these?

~~~
jacquesm
"Built in collaboration with our ODM partner QCT (Quanta Cloud Technology),
the current Big Basin system features eight NVIDIA Tesla P100 GPU
accelerators. These GPUs are connected using NVIDIA NVLink to form an eight-
GPU hybrid cube mesh — similar to the architecture used by NVIDIA's DGX-1
system. This setup, combined with the NVIDIA Deep Learning SDK, utilizes this
new architecture and interconnects to improve deep learning training across
all GPUs.

Compared with Big Sur, Big Basin will bring us much better gain on performance
per watt, benefiting from single-precision floating-point arithmetic per GPU
increasing from 7 teraflops to 10.6 teraflops. Half-precision will also be
introduced with this new architecture to further improve throughput."

~~~
iDemonix
So I just run the setup CD, right?

~~~
jacquesm
Assuming your question is serious: Yes, essentially, but you're going to have
to temporarily connect some peripherals if you don't intend to pop in a pre-
installed image on a drive. And of course, depending on what OS you intend to
run on it you might end up with driver hassles, your best bet is likely a
reasonably modern linux distro.

