
HP Updates Z8 Workstations: Up to 56 Cores, 3 TB RAM, 9 PCIe Slots, 1700W - jseliger
http://www.anandtech.com/show/11838/hp-updates-z8-workstation-up-to-56-cores-3-tb-ram-9-pcie-slots-1700w
======
the_real_sparky
I've got an HP Z640 workstation with dual E5-2690 v4's and it is basically
silent under full load. It has no problem running all 28 cores at 3.2GHz which
is the max all-core speed for that processor.

~~~
closeparen
Who needs something like this? What is your job? What do you use it for?

Really curious. I deploy code to a large production server cluster, my friends
in academia submit code to large scientific computing clusters, but I don't
know of anyone with this much power in their own desktop. I guess I've been in
video edit suites with machines much more powerful than the average PC, but
not this crazy.

~~~
mugsie
My usage is for building IaaS software. With a machine like this, I can have
VMs with VTX passthrough turned on, and simulate a cloud, without a rack of
physical servers.

As a remote employee it allows me to most of my development locally, and then
use larger environments for a shorter period of time later on.

Combine 3 or 4 of them, and I can have an actual cloud running under my desk,
for testing things like kubernetes deployments, with enough capacity for a few
concurrent test environments.

As I said above they run basically silent, so I can use them in a shared
office space without annoying my neighbors.

~~~
closeparen
Interesting, thanks for sharing. My instinct would have been to rent a few
dedicated boxes from a OVH for something like that. What made you decide to go
with local boxes?

~~~
mugsie
Personally, I find local dev much easier.

I had access to racks of hardware to do a lot of testing but they were in
Colorado, while I am in EU which made latency a real problem.

------
dis-sys
I just built myself a dual E5-2696v4 workstation last week. ebay/taobao has
loads of very affordable E5-2696v4, they are not ES/QS junks but the official
version provided to manufacturers. I paid $1,000 for each processor.

Supermicro X10DAI motherboard is just a piece of art, as usual. They sell you
this brand new beauty for $350 - it is more like running a charity given the
fact that those so called "high end" consumer grade motherboard with one
socket and 8 DIMM slots can easily cost you $500 or more.

RAM is pretty expensive now, 128G DDR4-2133P ECC REG RAM cost around $1,000 on
taobao.com, unless you buy some shitty brands to save a couple hundred $. Then
you need a EEB case, a solid reliable PSU and two heatsinks, that is another
$250 to spend.

I can't afford a 3T RAM HP workstation, but the good news is every programmer
in fact can afford a 44 cores dual E5-2696v4 workstation like the one I just
built. ;)

~~~
sireat
How's the buying experience for computer hardware on taobao?

I bought everything for a dual Xeon 2670v1 system (starting with Dell T5600 as
base) from ebay and it came out to $1000 last year including $150 for 64GB of
ECC RAM.

Very happy with it but could definitely use an upgrade now.

Dual E5-2696v4 seems like the current sweet spot for DIY.

~~~
dis-sys
> How's the buying experience for computer hardware on taobao?

pretty happy with my experience. vendors posted the stuff on the same day,
delivered within 24 hours as I live in Shanghai. it might be different story
if you live in other countries as the refund policy says you can return the
stuff within 7 days with no questions asked, you probably can't do that within
the 7 days time frame if the package need to be posted overseas.

> I bought everything for a dual Xeon 2670v1 system (starting with Dell T5600
> as base) from ebay and it came out to $1000 last year including $150 for
> 64GB of ECC RAM.

I also have a dual E5-2670v1 system, actually I am posting from it now. I
found a single E5-2696v4 is going to be 40-50% faster than the dual E5-2670v1,
dual 2696v4 almost triples the CPU performance for my workload. this has been
confirmed by both cinebench score and benchmark results of an in-house
application I developed. RAM bandwidth is not going to be a big jump.

> Dual E5-2696v4 seems like the current sweet spot for DIY.

Dual E5-2696v3 with some firmware/bios hack to push all cores running at
higher speed is probably more $ efficient. You can get a faster system that is
$500 cheaper. ;)

~~~
sireat
Makes sense that Taobao would be a great option in Shanghai :)

I wonder how easy it is to receive Taobao merchandise in Europe. It appears
you have to go through an agent.

------
lucaspiller
If the prices of one of these new are a bit out of your budget, but you still
want a relatively powerful workstation, take a look at used machines from a
few of generations ago.

Last year I picked up a Dell T3600 with a Xeon E5-2670 (V1 - 8C/16T) and 32GB
ECC RAM for €400 delivered. It's not completely silent (maybe HP are better in
that regard based on comments here?), but it doesn't make much noise - at
least compared to what you'd expect from the power it gives.

I now work from home, and having previously primarily used laptops, this thing
feels like a beast.

~~~
dis-sys
> a Xeon E5-2670 (V1 - 8C/16T) and 32GB ECC RAM for €400 delivered.

that stuff was $150-200 delivered if you buy from taobao.com.

people also need to realize that dual e5-2670v1 has a cinebench score of
2,000, that is just slightly highly than a single ryzen 1800x.

~~~
zokier
And last I checked there isn't a single Ryzen based workstation on the market,
something I wouldn't expect to change anytime soon.

------
scott_karana
If I wanted to build something like this myself, are the motherboards fairly
commodity? Or do you get into bizarre non-ATX custom form factors with bespoke
cases and PSUs to match?

I've built plenty of gaming rigs, but have never been interested in this world
until recently... and would appreciate any insight :)

~~~
oso2k
There's posts here on HN and I think techspot that discuss this [0][1].

[0] [http://www.techspot.com/review/1155-affordable-dual-xeon-
pc/](http://www.techspot.com/review/1155-affordable-dual-xeon-pc/)

[1] [http://www.techspot.com/review/1218-affordable-40-thread-
xeo...](http://www.techspot.com/review/1218-affordable-40-thread-xeon-monster-
pc/)

------
fdchn2016
Waiting for
[https://www.raptorcs.com/content/TL2WK2/intro.html](https://www.raptorcs.com/content/TL2WK2/intro.html)
to become available and prices to then fall over a year's time.

------
sytelus
I've been using Z840. Great developer machine for deep learning and heavy CPU
+ GPU related stuff. One of few workstations that had enough power to put in
two (old) TitanX in its days. Newer one looks even better.

------
userbinator
It will be quieter than a rackmount server since there's more room for bigger,
slower-turning fans that move the same rate of air, but at full load is still
likely to be quite loud.

~~~
uoaei
Put it in the other room! We have long enough cables for the peripherals.

~~~
PascLeRasc
I hope you don't need USB 3 for anything then.

~~~
icelancer
Powered extensions work just fine, I use 10m active cables with no issue.

~~~
katastic
So we're banking on a computer running with wires across into another room.
... or holes cut in the wall.

Instead of just buying a quieter computer (or upgrading it)?

~~~
uoaei
Seems like a simple, cheap solution compared to shopping around for a quieter
computer or parts.

How is it so problematic to have cables running along the molding and around a
corner?

In fact, why bother worrying about the noise anyway? Some people function much
better with white noise in the background.

------
desireco42
Can someone explain me why memory is costing so much? And would it make sense
to buy memory separately?

I am kind of playing /w configurations to see what I can get.

~~~
yuhong
The TSV technology that the 128GB DDR4 LR-DIMMs uses is still very new
(Samsung is the main vendor at this point). It is only recently that the price
of 64GB LR-DIMMs are getting close to the price of 32GB RDIMMs. As a side
note, you also need special Xeon models to get the full 1.5TB per socket with
the 128GB DIMMs now.

~~~
desireco42
I see. So essentially it would make sense to get one with smaller memory and
buy more when price goes down. Thank you for explaining.

------
rl3
> _AMD Radeon Pro WX 9100 Graphics (16 GB HBM2)_

Surprised the Radeon Pro SSG isn't an option. It's basically the WX 9100 with
2TB of flash memory slapped on that you can use as VRAM:

[https://pro.radeon.com/en/product/pro-series/radeon-pro-
ssg/](https://pro.radeon.com/en/product/pro-series/radeon-pro-ssg/)

~~~
DiabloD3
Not quite.

It requires special application support, you are 100% unlikely to ever
discover an app in the wild that can use it.

~~~
whyenot
According to the SSG website, Autodesk Maya will fully support the SSG,
including the 2TB memory.

"ProRender plug-in for Maya has full support for the out-of-core rendering
feature found on the Radeon™ Pro SSG which allows the application to tap into
the full 2TB of memory found locally on the card. This allows users with very
large models to render fully accelerated by the GPU. "

~~~
rl3
Apparently application support is implemented via the SSG API. It doesn't look
like it's a public API; I can't find any documentation anywhere.

------
elorant
Just for the RAM you'll need $20k. So I guess the whole rig would cost at
least $30k. That is of course if you max out all the equipment.

~~~
the_real_sparky
HP prices are... high. It's easy to spend $10k in processors alone for a dual-
Xeon system. The ones mentioned in the article will be over $13k each:
[https://ark.intel.com/products/120498/Intel-Xeon-
Platinum-81...](https://ark.intel.com/products/120498/Intel-Xeon-
Platinum-8180M-Processor-38_5M-Cache-2_50-GHz)

~~~
yeukhon
If you are talking about srlf-build, there is reason to avoid that for people
who can afford. It is called warranty. Hardware rig like this goes through
testing and the support package usually is pretty good. I don't have first-
hand experience, but I know Oracle would send engineers onsite to troubleshoot
if required. Unless you are ready to tackle the build, the testing, and the
support of the hardware rig, you want to avoid. You don't have to be Dropbox
or Netflix (I don't believe they build custom servers like facebook and Google
does, perhaps they buy from other vendors or from opencompute project).

~~~
PaulRobinson
I believe both Dropbox and Netflix are very, very cloud (specifically AWS),
heavy.

~~~
yeukhon
They are, but they have their own boxes for critical infrastructure. Dropbox,
however, moved a lot off of AWS.

------
myrandomcomment
But how many FPS with Minecraft? Can I play Civ6 on a huge world without
waiting forever...

Tounge firmly pressed in cheek..

~~~
katastic
I know you're not serious but Minecraft is single-threaded. So ... run 56
games at once?

~~~
myrandomcomment
1.8 implemented multithreading.

------
desireco42
I don't need it, but I would love if I could afford one and use it daily.

And those who comment who use them, just so you know I am jealous of you :).

~~~
dingo_bat
Same here. I'd use mine to just do regular stuff my laptop handles right now.
But I'd know I'm browsing facebook on a freaking supercomputer. And that'd
make me happy :P

~~~
KGIII
I've purchased whole racks full of servers that didn't have as much compute
power and cost a heck of a lot more than this. I use a nice mobile Titan but
it is nothing compared to this.

What crazy times we live in. I can't justify buying one of these. I keep tying
to come up with a good reason. I suppose YouTube would finally run smoothly.

------
bitmadness
Anyone here have experience regarding Dell or Lenovo workstations? Which
should I get?

------
akhilcacharya
What exactly are these machines used for?

~~~
oso2k
CAD, 3D workloads, 4K/8K video editing, stuff like that. They're great for
running dozens of VMs especially if you're running with SSDs and a non-
integrated RAID card. The integrated RAID chip on the z820/z620 is horrible.

~~~
akhilcacharya
I thought people used OSX/Win for video editing?

------
majidazimi
Atom/Electron devs: Challenge Accepted.

~~~
katastic
I can finally load the CNN webpage without adblock AND use a 30MB Excel
spreadsheet at the same time!

------
0xbear
These days if you need proper compute, you need to focus on the GPU. NVIDIA 10
series GPUs are 10x the throughput of high end intel CPUs, and less than
1/10th the cost. It’s not even funny anymore. It’s a shame you can’t even have
4 GPUs in such an expensive and bulky workstation. Back to the drawing board,
HP.

~~~
mugsie
Very few applications can make use of such a large amount of GPU cores - and
all of them would need much more CPU cores than this provides to coordinate
the work.

Any applications that can would be either run on clusters of servers to really
parallelise the work, or is so custom you may as well buy a threadripper and a
custom case.

Hell most apps can barely take advantage of multi core CPU.

~~~
jeff_friesen
I somewhat disagree. Maybe there are few compiled applications that can use
those resource, but it's easy to write code that uses all of them.

I have a 40 core CPU machine with a GTX 1080 Ti GPU. I run deep learning
models with 90% GPU utilization and those 40 cores are barely used.

I would love to have 3 more GPUs to run in parallel to test different neural
network architectures. Sometimes I'll run a CPU script at the same that
processes machine learning data that uses all 40 cores.

I would use 4000 CPU cores and 10 GPUs in a machine if I could get them and I
don't even do machine learning full time. I'm personally happy to see this
trend of more core counts.

~~~
mugsie
This is my point - I dont think it is as easy as you think to scale from 3.5k
cores in a 1080ti to 14k ish cores.

Sure you can write code that nominally use all the cores, but I do not think
that the performance increase is going to be linair to the core count.

It's not even down to CPU core count - it would be limited by the speed of a
single core.

