
Assessing Cavium’s ThunderX2: The Arm Server Dream Realized - drewg123
https://www.anandtech.com/print/12694/assessing-cavium-thunderx2-arm-server-reality
======
thisisit
An important side note here is that Cavium has been bought by Marvell
Semiconductors:

[https://in.reuters.com/article/cavium-m-a-marvell-
technlgy/m...](https://in.reuters.com/article/cavium-m-a-marvell-
technlgy/marvell-technology-to-buy-rival-chipmaker-cavium-for-6-billion-
idINKBN1DK1TL)

So future of ThunderX2 remains to be seen.

~~~
wtallis
I don't think Marvell has anything that overlaps with ThunderX2, but they
definitely have a lot of complementary products. I don't think they have any
reason to cancel ThunderX2 if it looks like it can be successful.

------
alberth
I thought TDP was a selling point of ARM for servers.

Cavium has a higher TDP (180W) than either of the Intel Xeon's listed in the
article (150W, 165W).

Is this more about "performance per watt" then?

EDIT: Also makes me wonder just how much more energy efficient and/or
performant Cavium would be if they moved away from TSMC 16nm fab over to
Samsung 10nm fab (like Qualcomm). There's a sizeable difference between 16nm
vs 10nm (assuming the fabs are using the same metrics). Qualcomm using Samsung
10nm fab is only 120W.

~~~
jandrewrogers
I know many people that have evaluated ARM for a variety of server
applications and ARM's advantages, in terms of tradeoffs, are not what many
people assume. Where ARM shows well is applications that spend a significant
amount of time idle in power-restricted environments (battery-powered
analytics clusters being a common example) and running software that is weakly
optimized for efficiency (some web app stacks look like this). In practice,
this is a niche set of requirements for servers.

The challenge that ARM has long had in servers is that _highly optimized and
efficient_ Intel codes have competitive performance per watt and performance
per dollar. Cavium is quite open that their cores are optimized for low
ILP/IPC software, which is essentially arbitraging software that is poorly
designed for x86 silicon. As data and analysis scales have increased, more
standard server software is specifically designed to be hyper-efficient on
Intel, not just HPC codes like the old days. This makes it increasingly
difficult for ARM to compete with the operation throughput per watt/dollar
that is possible with Intel and well optimized software.

There is the additional issue with platforms like ThunderX2 that even though
it could be competitive in theory, it would require software to be
specifically engineered around the peculiarities of the microarchitecture.
Software that assumes Intel microarchitectures and optimizes toward that,
either explicitly or implicitly, will inherently be suboptimal on ThunderX2.
This isn't something that can be fixed in a compiler, it is intrinsic to the
architecture of the software at a higher level e.g. C++ codes embed many
assumptions about CPU cache topology and properties.

~~~
moconnor
I worked in HPC for over a decade. Most of the codes are not hyper-efficient
on Intel - particularly not current-generation Intels that need efficient use
of very wide vectors to reach peak performance.

Many codes are memory-bound, and the TX2 has excellent bandwidth. This shows
up in real-world simulation codes such as OpenFOAM.

~~~
auvi
Very interesting as I used to do OpenFOAM parallel runs years ago on x86. Do
you have any links to OpenFOAM benchmarks on ARM64?

~~~
Ar-Curunir
The conclusion of the linked article has a reference to a relevant paper

~~~
gnufx
But, as usual, you don't know what the profile of the calculations were, in
particular because you don't know the mode it's operating in/data it's
operating on. ("OpenFOAM" is actually many different programs.)

That said, the indications are that the performance is decent for HPC. What I
haven't seen is a comparison with Ryzen (or POWER9). There's also a lack of
data even on the SIMD hardware in ThunderX2 and POWER9, at least that I've
been able to find.

------
notacoward
Between this and Zen, Intel is definitely starting to feel some heat. The next
year or so is going to be interesting. Maybe Intel will recover; maybe we'll
look back and see this as the point where they lost their grip.

~~~
axaxs
I used to think this, but really don't see it happening. Intel has a lot of
weight, money, and is pretty vertically integrated. They've been rather happy
to stick where they're at, where the money is good. And no company is really
threatening them there, and may not in the foreseeable future. But make no
mistake, once that market begins to erode, they'll branch out more. Their
mobile efforts have been mostly half assed, but I have no doubt they could
cream both Apple and Qualcomm if they threw their resources at it.

------
alberth
Can anyone speak to the techical merits (or disadvantages) of arranging your
Cores by Mesh or Ring?

I noticed all the ARM chips use a Ring bus whereas Intel uses a Mesh.

~~~
Symmetry
Intel was using a ring too until their most recent architecture. Sometimes
multiple intersecting rings when the number of cores grew too large.

------
auslander
If it could Libreboot, that would be like first free of firmware blobs
practical server in decades.

Good Marvell bought it. Their ARM boards are already fw blob free.

~~~
spacenick88
Have a look at Talos II, it's using POWER9 with completely open firmware

~~~
auslander
Checks out, I stand corrected, thanks. libreboot.org/news/talos.html

------
newprint
I would curious, where can I buy this system ? Anyone one know resellers ? At
$$$ listed, I would buy cheapest system in a heartbeat.

~~~
rwmj
Cavium are actually pretty good out of all the ARM server vendors of, you
know, selling the hardware. (For some reason other vendors found that really
hard.) The ThunderX2 has not been generally released yet, but I'm expecting
it'll turn up on the same sites selling the old ThunderX, eg.
[https://www.avantek.co.uk/store/arm-
servers.html](https://www.avantek.co.uk/store/arm-servers.html)

~~~
aseipp
Out of curiosity I looked, and they actually already have pricing for the
ThunderX2 Workstation from Gigabyte -- the very one mentioned in the article:
about 10,000 GBP out of the box.

[https://www.avantek.co.uk/store/avantek-thunderx2-arm-
workst...](https://www.avantek.co.uk/store/avantek-thunderx2-arm-workstation-
thunderx2station.html)

(They have ThunderX2 blades on the server section of their store with no
prices yet, however.)

------
pmontra
I know ARM processors shouldn't be subject to the recent vulnerabilities, or
not as much as Intel and AMD, but I'm surprised that the article doesn't
address that point. A decent CPU not vulnerable to Spectre, Meltdown and
friends should be able to sell well only because of that.

~~~
floatboth
The ISA doesn't matter very much.

Meltdown is specifically Intel's fault.

Spectre is, AFAIK, inherent to out-of-order execution. So the original
ThunderX is not vulnerable, ThunderX 2 is.

------
monocasa
> branch intensive code (databases, AI...)

AI is branch intensive? Like, video game style state machines are, but that's
not what people mean these days when the say AI in connection with servers,
right?

------
yazr
Can i get the SPECjbb benchmark mentioned in the article ?

The web site claims that it is pay-only $1500
[https://www.spec.org/order.html](https://www.spec.org/order.html)

------
jsgo
sounds great, didn't I read that ARM was backing off on server though (hoping
it was a bad article).

edit: thanks, you all nailed it. I remember now and it was Qualcomm. Whew, I'm
glad as I was bummed that this meant there was a good step for a piece of tech
that would be DoA. Really wish Qualcomm wasn't abandoning based on how it
appeared they were making progress as well to be a good competitor (and
competition has been great where applied).

~~~
mtgx
Not Arm, but Qualcomm. It was also a strange and _sudden_ decision. I don't
think they really wanted to do it, but might have been forced to do it by the
circumstances: anti-trust lawsuits, Apple dumping them, Broadcom attempting a
hostile takeover, U.S. government pressuring them not to sell to Broadcom no
matter what, etc.

One or all of these may have forced Qualcomm to cut its losses. Either way, I
hope they get to sell the Centriq line to some other company that can provide
competition to both Cavium and Intel/AMD/IBM in the server space. We need more
than one Arm chip to compete in the space, otherwise Cavium's chip could also
be seen as some kind of outlier and be ignored by the industry.

~~~
wyldfire
> One or all of these may have forced Qualcomm to cut its losses.

Yeah, I think the LBO really scared Qualcomm. Their royalties fund a lot of
the other work in the company and I think they've learned that it also makes
them a target. A domestic LBO could happen and then there'd be no escape
hatch. They pledged to cut costs when Broadcom announced their intentions,
this may be a part of that.

I don't think of them selling the server chip business as "cutting its losses"
\-- instead I would call it narrowing its focus back to businesses closer to
its core competency. The Centriq launch with Amberwing seemed to be relatively
positive for a first design release. They didn't turn the industry upside-down
but they probably achieved most of what they set out to do. It will take some
time to convince server customers to switch, recompile their code and get
their vendors to recompile their code.

> We need more than one Arm chip to compete in the space, otherwise Cavium's
> chip could also be seen as some kind of outlier and be ignored by the
> industry.

Agreed. There's momentum that's been built [1,2] and I'd hate to see it flub
now.

[1] [https://www.redhat.com/en/blog/red-hat-introduces-arm-
server...](https://www.redhat.com/en/blog/red-hat-introduces-arm-server-
support-red-hat-enterprise-linux)

[2] [https://www.suse.com/c/news/suse-steps-up-to-support-
innovat...](https://www.suse.com/c/news/suse-steps-up-to-support-innovative-
arm-solutions-for-customers/)

~~~
jsgo
I hope they do something with it though. Whether sell it to someone who can be
a good caretaker to it, share lessons and designs with other companies that
could then build upon it, spinning it off, whatever they choose. Just
abandoning it outright seems like an unfortunate move. I won't say bad,
because I'm sure they have their reasons, but unfortunate in that it seemed
like it was on track to be viable.

Microsoft's Surface Pro line, to me, was kind of an "eh, that's nice I guess"
up until the Pro 3 at which point it seemed like a platform that had hit its
stride. The stuff I was reading on Amberwing seemed like it wasn't a silver
bullet, but what it did work for, it worked very well when comparing price vs
performance so one could hope that with time, they could make it even better.

