
Cloudflare CEO: Why we’re switching to ARM-based servers - Aissen
https://twitter.com/eastdakota/status/976560820611031040
======
Aissen
A blog post is coming, so I might have jumped the shark here. An interesting
note from a Cloudflare engineer:

 _Most powerful Xeon is the 28 core Platinum 2180 at $10k RSP and >200W TDP.
Due to the nature of Intel turbo boost it almost always operates at TDP under
load. Our ARM is the Centriq 2452 at less than $1400 RSP, 46 cores and 120W
TDP, that it never hits. Beats Xeon 9/10 workloads. _

[https://twitter.com/thecomp1ler/status/976617883164921857](https://twitter.com/thecomp1ler/status/976617883164921857)

~~~
dvfjsdhgfv
This is amazing! We're witnessing the beginning of the demise of Intel in the
data canter, right in front of our eyes. I had thought this would take
decades...

I just hope that thanks to large companies like CloudFlare buying large
numbers of ARM servers their prices finally start going down to a reasonable
level.

~~~
sigi45
I don't think so. I'm guessing that the xeons feature set is not used at all
and this specific payload is just much better on that arm.

There is a big chance, that whatever cloudflare is doing on that machine, is
massive parallel and simple.

~~~
pdpi
My understanding is that some large parts of the feature set aren't usable
anyhow. E.g. there's enough of a ramp up/ramp down power cost for the vector
units that you'd never want to compile your binaries with SSE/AVX turned on
unless your workload makes near constant use of those units.

~~~
sigi45
I have never heard that one before. Do you have any sources for that?

~~~
Jenya_
[https://www.quora.com/Why-are-AVX-instructions-so-power-
hung...](https://www.quora.com/Why-are-AVX-instructions-so-power-hungry)

[https://news.ycombinator.com/item?id=13925039](https://news.ycombinator.com/item?id=13925039)

~~~
pdpi
Thanks for the quotes. Apparently I slightly misremembered the problem. From
the second one, this is what I was thinking about:
[https://news.ycombinator.com/item?id=13926839](https://news.ycombinator.com/item?id=13926839)

------
walrus01
Mid priced xeons: all sorts of motherboards available from eight different
Taiwan based OEMs.

AMD: I need some threadripper or server Zen motherboards, again there's 4+
OEMs with many designs.

Both amd and Intel: all sorts of stuff available that meets open compute
platform dimensions for motherboards. Or for ordinary 1U servers, or atx size
boards that can be mounted in DIY blade enclosures.

ARM: where are the motherboards I can buy right now in small to low
quantities, and what are their specs/price? Who makes them? It I want to order
40 or 400 tomorrow... where do I go?

Edit: I just spent ten minutes googling for arm centriq platform motherboards
and cannot find a single product I can actually _buy_. As compared to how
easily I could start the process of ordering a hundred 1U xeon based servers
in the next half hour.

At a massive scale such as cloudflare, sure you can afford to custom order
thousands of motherboards. But for a lot of operators who are not nearly as
big as them, the existing economies of scale in the x86-64 platform are a
greater advantage. Right now I'd be willing to bet that dual socket current
generation xeon motherboards are manufactured in much greater numbers, as much
as a ratio of 500:1, compared to ARM server motherboards. Being able to mix
and match whitebox parts from Taiwan and China sources to DIY bare metal blade
hypervisors is a huge advantage.

~~~
majewsky
Also, when you can get your hands on a ARM motherboard, will you be able to
boot mainline Linux or only some ancient kernel version that the vendor has
patched with their own binary drivers?

~~~
imtringued
Servers don't need a Mali GPU which is the major bottleneck to support
mainline linux.

I'm constantly amazed by people wanting to see Intel die for a minor price
decrease. The open source drivers for intel gpus might have not been provided
by intel but at the very least they exist. The same cannot be said by ARM.
When it comes to proprietary software ARM is at least as evil as intel. By
replacing intel with ARM you're going from bad to worse.

~~~
walrus01
Agreed, and Intel actually had a track record of really good open source
driver support for important stuff that is not GPU. Example, 11+ years ago
when a single 10 gigabit NIC cost $6000, Intel had their own staff write and
submit freebsd and Linux kernel drivers for their first gen card. The same has
continued up until now if you look at the Linux kernel commits for Intel
network interfaces.

------
robeastham
Saw that the Centriq 2434 (40 cores) is listed at RRP $888.

Does anyone know if you can get hold of a Centriq cpu & compatiple motherboard
to build a local server or two? I fancy one in our lab rack to play around
with.

I'm assuming that this is not possible and that they are currently just for
datacentre partners who make a large order. But would be great to hear
otherwise.

~~~
bjackman
Not Centriq but I think there are boards with Cavium ARM chips that you can
buy as a consumer (can't find link to cite so may be wrong, but may be a
useful avenue for your search).

------
jbarham
Will be even more interesting when RISC-V becomes a contender in the HP space.

~~~
ZirconiumX
I don't know why this is being downvoted.

One of the workloads that Cloudflare does is encryption, and RISC-V has been
planning a vector extension that can be reconfigured, so that the jump from
128-bit to 256-bit native execution is not an ISA change and doesn't require
recompiling.

~~~
imtringued
Doesn't every CPU manufacturer do this?

AMD offers 256 bit vector instructions but they take twice as long because the
Zen cores can only process 128 bit at a time.

------
Algent
Since we are speaking about CF, they usually a do lot of crypto. Which, unless
I'm wrong, is accelerated by a lot on Intel CPUs.

I'm really curious at what the blog post will reveal on this.

~~~
Deathmax
There is a blog post from CF a few months back with benchmarks.
[https://blog.cloudflare.com/arm-takes-wing/](https://blog.cloudflare.com/arm-
takes-wing/)

------
walrus01
For a bit more on what the manufacturer reference design looks like:
[https://www.nextplatform.com/2017/12/06/deep-dive-
qualcomms-...](https://www.nextplatform.com/2017/12/06/deep-dive-qualcomms-
centriq-arm-server-ecosystem/)

------
jakozaur
Not surprised given:

1\. A lot of cloud services are abstracting CPU architecture. E.g. you don't
care on which CPU AWS S3 runs.

2\. ARM is using less power than x86.

3\. ARM license fees are so low comparing to Intel high gross margins. Whole
mobile world runs on ARMs and little profit is made on CPUs there comparing to
Intel cash cow server chips.

~~~
ofrzeta
1\. Is it? Actually I do not want to recompile all software. Did you try
running Docker or Kubernetes with ARM? It's there but there's not a whole lot
of software/images you can run.

~~~
exikyut
Software follows hardware. Software is just commentary, particularly throwaway
software.

