
ARM server market - Twirrim
http://perspectives.mvdirona.com/2015/10/arm-server-market/
======
rbanffy
One crucial point when we talk about high core count parts is memory and IO
bandwidths. In the end, that'll determine the overall performance of your
server. I don't see ARM parts, due to their relatively low single-thread
performance, as good for compute-bound workloads, but, with memory and IO
channels significantly faster than current x86-based parts, they could be
competitive in IO bound ones.

I'd love to see more diversity in the server space, but reality is harsh.

~~~
kev009
This is where IBM cleans shop. POWER is pretty mean when it comes to
bandwidth. You pay the tax to be a couple years ahead of the Intel curve if
you need it.

~~~
rbanffy
Indeed, but there is no magic there. If ARM-based servers with high core count
can be made with bandwidths in the same league as POWER, IBM's high-margin
server business may end up being threatened.

Also, we cannot forget POWER is very good with single threaded performance.

------
rwmj
And finally a standardized platform, so you can just install any commercial
Linux distro on any hardware:

[https://www.arm.com/about/newsroom/arm-ecosystem-
collaborate...](https://www.arm.com/about/newsroom/arm-ecosystem-collaborates-
to-deliver-initial-server-platform-standard.php)

No more random fork of uboot, proprietary boot method and guess-the-device-
tree. Instead you get the goodness(?) of UEFI, ACPI and standard minimum
hardware.

~~~
wyldfire
Are there any breadboard-style releases implementing this spec? I'm looking
for the low(er) cost and open ecosystem that I find in ODROID and RasPi style
boards, but still the robustness and standardization of this common spec.

~~~
rwmj
There's a new development board coming real soon that will implement it. It is
also implemented by the firmware on APM X-Gene (Mustang) and HP Moonshot. Not
sure about Cavium as I've not used it.

However it's unlikely you'll find a small format SBSA-compliant board any time
soon, because the server stack simply assumes a lot more RAM than is available
on phone/tablet SoCs. For example, the minimum RAM required to run
RHEL/aarch64 is 1 GB/pCPU (so in reality 4-8GB min), and even the Snapdragon
808 only has 3 GB. I personally wouldn't be happy doing development work for
64 bit ARM SBSA with less than 8 GB of RAM, and for OpenStack, 32 GB is the
minimum I'd recommend.

(I do run Fedora/aarch64 on my LG G4 phone though :-)

------
apendleton
I think server-class ARM is interesting, but it's always struck me as much
more interesting for on-premises stuff... like, you want a cheap on-site NAS
that doesn't use much power, or a generic machine you can use as a switch, or
maybe even an intranet host or something. Anything in the category of stuff
that is always on but not very resource-intensive.

In cloud deployments, though, sharing a bigger box amongst lots of virtual
machines just seems way more economical.

~~~
esaym
I kept a close eye on the ARM server market for years waiting for it to happen
and bring cheap multi core boxes to the masses. Ultimately, I stopped caring
when quad core Xeon boxes got down to less than $400 and draw less than 30
watts idle: [http://amzn.com/B00FE2G79C](http://amzn.com/B00FE2G79C)

~~~
skrause
I still think that 30 watts idle are way too much for at-home equipment. At
German electricity prices we're still dealing with $75 a year just for
electricity. What I want to see is a reasonably powerful machine that is <5
watts idle (for the board, CPU and RAM) plus the power of the installed hard
disks.

~~~
creshal
Right now you'd need NUC-style nettops for that kind of wattage. I'm not sure
the performance qualifies as "reasonably powerful", though…

------
ksec
I am going to presume, for ARM to win any market share, they will have to
offer a competitive advantage. ( Correct me if I am wrong )

So where does ARM's Server Chip actually offer an Advantage?

Price - The CPU is only part of the Server's Cost. When you look at the whole
picture, The Memory, I/O Controller & SSD, Network, will all cost the same.
The higher price of the memory and SSD / HDD, the lower total cost % of the
CPU. And once you lowered to a certain percentage, you want an incredibly
cheap, Atom Server CPU is available for less then $50.

Performance - Even in the chance that any ARM's Server CPU market manage to
make a competitive CPU core against Xeon, which in itself is an incredible
achievement, they will have to also compete against Intel's world class ECC
Memory Controller, Network Controller, and I/O controller. All three are the
main reason AMD CPU didn't offer any competition even when they are much
cheaper.

Power - This is similar to Price, once you factor in the memory and I/O, CPU's
role is relatively small. And the power / energy usage scenario on a server
much flavor Intel's rather then ARM.

So where does it offer a advantage? When you want a small baremetal server
that is cheap, but in the world where VM is a common place, there is no reason
why you cant have a small instance of it running on a much larger CPU. And
even Intel admit it, none of their Server Customers, ( Read NONE ) wanted the
Atom as they thought. It turns out everyone wanted Xeon-D, you pay a little
premium for HUGE amount of flexibility. Intel wanted the Atom to disrupt
itself rather then ARM, and it turns out it was the Xeon-D that did that.

~~~
rphlx
The current batch of ARM microserver parts are indeed pretty primitive
compared to x86, but each year they will copy more & more of the tricks and
optimizations that Intel implemented 10-15 years ago, including the superb
virtualization tech. Still I agree they are most likely going to fail due to
the inevitable patent infringement lawsuits, the fab gap, and/or customer
reluctance WRT deploying ARM SW.

------
steckerbrett
I thought ARM servers might be nice on scaleway but was quite disappointed in
the performance. 32bit and all disk IO over a network connection is really not
a good start. Maybe that will improve when we see more ARM64 around.

~~~
hyperbovine
At .004 euro cents an hour, seriously, what were you expecting? 32 bits seems
like they threw in 16 bits for free.

~~~
steckerbrett
It's approximately the same price as Digital Oceans lowest offering, which has
significantly higher performance in every area.

~~~
dbaupp
Are you (and parent) looking at something other than their pricing pages? It
seems to me that scaleway costs 0.6 euro cents/hour, but also generally has
higher numbers for most features than Digital Ocean: more cores, more RAM,
more disk, more bandwidth.

[https://www.digitalocean.com/pricing/](https://www.digitalocean.com/pricing/)
[https://www.scaleway.com/pricing/](https://www.scaleway.com/pricing/)

~~~
steckerbrett
My comparison was between the C1 offering from Scaleway and the lowest
DigitalOcean package. Scaleway has far larger disk and memory, but the disk is
remote over the network link and the 4 ARM cores are 32bit and terribly slow.
I really like Scaleways configuration and control panel but the performance
just isn't there. It's likely better for a lot of things other people are
doing but I needed the IO and CPu performance.

~~~
ju-st
How did you measure CPU performance? With DO you have propably a high "burst"
cpu performance but if you keep using a lot of CPU ressources they will
probably supsense your VM because of "abuse" or something (my experience with
other VM providers). On Scaleway you have worse CPU performance but you can
use the full CPU 24/7.

~~~
steckerbrett
Just general feeling compiling some packages and messing with 'openssl speed'.

~~~
4ad
Compiling software is a somewhat meaningful metric across architectures (with
caveats), but openssl speed is not really indicative of the CPU performance,
rather it's benchmarking its own fast assembly routines. That only matters if
your workload is crypto-bound.

~~~
rjsw
You could also be benchmarking the interface between openssl and crypto
hardware, several ARM SoCs have crypto accelerators.

------
samcheng
I've been out of the loop for a while datacenter-wise, and so fail to see the
immediate value here.

Are these servers running virtualized instances? Are they much cheaper as far
as $/request or watt/request (or other efficiency metric? How do they compare
to a slice of an Intel server, which has the benefit of a more mature
environment?

~~~
Twirrim
ARM has typically been more efficient, for both heat and power consumption,
while losing out on computational power.

That's enabled people to cram significantly more processing cores into a
server rack. 4 years ago HP release a server line with 288 quad-core ARM
processors in 4U, for a total of 1152 cores. Obviously that introduces the
complexity of needing code that is suited to massive parallelism. Each
processor there took up just 1.5 watts idle, 5 watts under load. By comparison
the Atom from Intel at the time had comparable performance, but consumed 8.5
watts, just over 70% more, and didn't idle to as low consumption.

There are a number of problems that are significant with large scale data-
centre operations, but power consumption and heat generation are way up there
at the top. In theory a data-centre filled with ARM based servers would give
you comparable performance, but with cheaper electrical and climate control
bills.

The emphasis there is on "in theory" :) The reality at the moment is that
companies have spent years trying to get ARM based, massive core count servers
to be a thing, and it hasn't really worked out. Switching processor
architectures is never something that can be done lightly. Software has to be
compiled and supported provided for it, and isn't necessarily tuned for the
ARM architecture. The x86 architecture's performance characteristics have been
really well understood, and software likely designed with those
characteristics in mind.

Historically ARM has also lacked Windows server support, and had very variable
quality linux support (not helped by every Tom, Dick and Harry SoC company do
the darnedest things in pursuit of creating 'value'.) As far as I've heard,
they've since done (or started on?) a huge restructuring of the ARM path in
the kernel to clean up the mess and provide significantly better customisation
opportunities.

In part it has also seemed like a case of "no one ever got fired for buying
IBM". You've only got a certain budget, do you gamble on an unproven (to you)
architecture, or stick the the tried and tested Intel?

~~~
nkurz
Twirrim: _ARM has typically been more efficient, for both heat and power
consumption, while losing out on computational power._

i_have_to_speak : _Current ARM server chips are much worse than Intel chips on
the performance /watt metric._

It's hard to square these two statements. It might depend on how you define
"efficiency". If efficiency is the amount of energy to perform a calculation,
I don't think ARM is more efficient. This paper from a couple years ago
concludes that ISA is no longer a defining factor:
[http://www.embedded.com/design/connectivity/4436593/Analysis...](http://www.embedded.com/design/connectivity/4436593/Analysis-
of-contemporary-ARM-and-x86-architectures)

Separately, this is an excellent article comparing recent generations of Intel
against each other, showing that although power use has been going up,
"instructions per cycle" has been going up even faster, resulting in a net
improvement in energy efficiency:
[http://kentcz.com/downloads/P149-ISCA14-Preprint.pdf](http://kentcz.com/downloads/P149-ISCA14-Preprint.pdf)

~~~
Twirrim
I wonder if the question we ought to be asking is "How idle is your server
farm"? ARM still uses significantly less power than Intel chips do when idle.
If your fleet is working hard all the time, it would be a no-brainer to go
with Intel. What if your fleet spends 50% of the time not working? 70%? There
presumably is some tipping point there.

~~~
bsder
Cloud services probably upended this.

If my server farm is idle any significant amount, I'm going to take those
servers offline. If I need capacity, I'll spin up some cloud instances in the
short term, and I'll bring my own servers online until I'm at an appropriate
idle/busy metric.

I suspect that the days of 99% idle servers are long gone. I suspect that
utilization is probably above 70%.

~~~
pjc50
The interesting market is the "personal" server, which is idle 99% of the time
but cannot be turned off as the owner only has 1 to start with.

~~~
discodave
That's not an interesting market because it is tiny and will get relatively
smaller going forward.

There are 2, maybe 3 interesting markets that will come to dominate computing
even more than they already do in the next few years:

Cloud, Mobile and IoT (maybe)

Desktops, personal servers and any other non-cloud server workloads will be a
rounding error.

------
ergo14
[https://www.hetzner.de/us/hosting/produktmatrix/rootserver-p...](https://www.hetzner.de/us/hosting/produktmatrix/rootserver-
produktmatrix-ax) \- this is an interesting offering, Hetzner is one of bigger
hosting companies in Europe. They seem to be experimenting with cheap ARM
offerings recently.

------
anjanb
Looking forward to a high performance JVM on the ARM server. Is there a way to
rent an ARM server which has a high-performance JVM ?

~~~
needusername
Oracle JDK has full support for C2 and hard float on ARM 64. That code is
_not_ in OpenJDK. Red Hat is working on C1 in OpenJDK for Java 9.

~~~
enevill
Full support for C1 & C2 is available in OpenJDK 7,8 & 9\. Prebuilt binaries
may be downloaded from
[http://openjdk.linaro.org/releases.htm](http://openjdk.linaro.org/releases.htm).
Sources are upstreamed for jdk9
([http://hg.openjdk.java.net/jdk9/dev](http://hg.openjdk.java.net/jdk9/dev)).
For jdk8 they are maintained in the OpenJDK aarch64 project
([http://hg.openjdk.java.net/aarch64-port/jdk8](http://hg.openjdk.java.net/aarch64-port/jdk8)).
For jdk7 they are maintained in the IcedTea project
([http://icedtea.classpath.org/hg/icedtea7-forest](http://icedtea.classpath.org/hg/icedtea7-forest)).

The default java on Ubuntu trusty (14.04) and vivid (15.04) LTS releases is
openjdk. From my 48 core aarch64 platform running trusty.

ed@arm64:~/jdk8/jdk8$ cat /etc/lsb-release DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu
Trusty Tahr (development branch)" ed@arm64:~/jdk8/jdk8$ uname -a Linux arm64
3.18.0-g57fcd51 #1 SMP Fri Jun 26 17:33:54 PDT 2015 aarch64 aarch64 aarch64
GNU/Linux ed@arm64:~/jdk8/jdk8$ java -version java version "1.7.0_51" OpenJDK
Runtime Environment (IcedTea 2.4.6) (7u51-2.4.6-1ubuntu4) OpenJDK 64-Bit
Server VM (build 25.0-b70, mixed mode) ed@arm64:~/jdk8/jdk8$

OpenJDK is also the default java on RHEL and fedora.

------
Tepix
I think these servers can be interesting for increased privacy. You can use
them as a low-cost, low energy dedicated server where noone can snoop on you
undetected (unlike a VPS).

------
eva1984
Until they succeeds to persuade AWS/GCE/Azure to sell ARM backed instances,
they will be taken seriously.

~~~
SixSigma
Let's not lose sight of who James Hamilton is :

From his Linkedin :

VP & Distinguished Engineer

Amazon.com

December 2008 – Present (6 years 11 months)

A member of the Amazon Web Services team. Specializes in infrastructure
efficiency, reliability and scaling. Prior to joining Amazon.com, James was
Microsoft Data Center Futures Architect. He has spent more than 20 years
working on high-scale services, database management systems, and compilers.

And a couple of biographies :

[http://highscalability.com/blog/2015/1/12/the-stunning-
scale...](http://highscalability.com/blog/2015/1/12/the-stunning-scale-of-aws-
and-what-it-means-for-the-future-o.html)

[http://www.wired.com/2013/02/james-hamilton-
amazon/](http://www.wired.com/2013/02/james-hamilton-amazon/)

~~~
AstroJetson
He presently lives on his trawler, he's presently on an island 1200 miles west
of Australia.

    
    
      http://blog.mvdirona.com/

------
RP_Joe
That link is dead.

~~~
tw04
His site doesn't appear to be able to handle the load. Google has it cached
thought:

[http://webcache.googleusercontent.com/search?q=cache:CAYvaWj...](http://webcache.googleusercontent.com/search?q=cache:CAYvaWjavB8J:perspectives.mvdirona.com/2015/10/arm-
server-market/+&cd=11&hl=en&ct=clnk&gl=us)

