
IBM Power System E980 – Up to 64TB RAM per node [pdf] - peter_d_sherman
http://www.redbooks.ibm.com/redpapers/pdfs/redp5510.pdf
======
abridgett
The redbooks are hidden gems - what documentation should be. When I used to
work with these (very nice) bits of kit, the redbooks were my bible.

They'd tell you the gory details, you could learn from the experiences of the
authors. The "official" documentation was good, but wouldn't ever say "you
don't want to do it this way, it'll end in tears".

PS: Note that the book is 2 years old now. And my experience with them closer
to 10 ;-)

~~~
abraae
It's hard to convey how different acquiring knowledge used to be in the old
days.

There was so much less information .. no Google, no online groups, and
certainly no stack overflow .. but red books were true compediums of hard won
knowledge.

Today the core skill is being able to drive Google, to quickly sift search
results, to parse through page after page of shitty low quality cruft to find
the nuggets of knowledge you need.

Back in the day, you could read a red book on a plane flight and land with a
greatly enhanced foundational understanding of the subject.

------
peter_d_sherman
(Page 3:)

>"The Power E980 server provides the following hardware and components and
software features:

Up to 192 POWER9 processor cores.

Up to 64 TB memory.

Up to 32 Peripheral Component Interconnect Express (PCIe) Gen4 x16 slots in
system nodes.

Initially, up to 48 PCIe Gen3 slots with four expansion drawers. This
increases to 192 slots with the support of 16 I/O drawers.

Up to over 4,000 directly attached SAS disks or solid-state drives (SSDs).

Up to 1,000 virtual machines (VMs) (logical partitions (LPARs)) per system."

~~~
api
They should definitely offer windowed cases, lots of colored LEDs, and neonz.

~~~
bargle0
You joke, but supercomputers often have elaborate, decorative racks and vinyl
decals.

~~~
nimbix
Isn't there a story about how a supercomputer used a macbook just to display a
nice the logo on the door - and for nothig else?

~~~
AtomicOrbital
as I recall a reporter quipped to Seymour Cray that Apple uses a cray to help
design the mac to which Dr Cray replied he uses a mac to design his
supercomputers

------
louwrentius
[https://yourdatafitsinram.net](https://yourdatafitsinram.net)

~~~
gnufx
It doesn't mean anyone will take any notice (or that it's feasible to get data
in and out). Long ago, supporting crystallography, I pointed out that each
image would roughly fit in the cache, and the whole dataset in memory. (On
Alphas with ~10MB and 1GiB, I think -- though the storage array was only
~250GB.) The developer of the processing program still didn't see fit to
replace the disk-based sort written for a PDP11, which was where the
bottleneck was.

------
crmd
64 TB is all that that anybody would ever need on a computer.

~~~
Koshkin
No, that's 640.

~~~
yjftsjthsd-h
...in fairness, I _do_ struggle to think what you could ever do to exceed
640TB of RAM, other than some very big scientific/simulation things (or SAP or
Google Chrome;]). I mean, that's more _persistent_ storage than I expect most
people to need for at least another decade (or at least whenever virtual
reality really takes off), let RAM.

~~~
rbanffy
A sufficiently complex immersive VR with enough resolution could take up that
kind of memory, but it's very much in the realm of science fiction right now.

When I'm dead and my mind is uploaded to one, a machine like this may prove
comfortable.

~~~
steffan
> a machine like this may prove comfortable

Future virtual real estate listing: "Spacious 640TB expanse gives you room to
let your mind wander. Large enough for two entities to share and even spawn
subprocesses!"

------
rbanffy
IIRC, you can get E980-based VMs on IBM cloud running AIX or i. Also, IIRC,
under PowerVM, these cores run 4 threads each.

------
zitterbewegung
People here seem to be struggling for what to do with 64TB of memory.

The obvious ones are fluid dynamics, weather simulations, n-body problems, and
simulation of quantum systems.

I know that Capital One gave a talk about they had an in memory anomaly
detection system for fraudulent transactions which was under 1 Terabyte.

A less obvious one would be to accelerate trading of NLP systems like training
GPT-3. You need the memory to not only keep the dataset and model you are
training in memory. NVIDIA seems to have this idea:

[https://top500.org/system/179397/](https://top500.org/system/179397/)

Also, you could clean data by just loading all of the data that you collected
in RAM and then cleaning it instead of swapping to disk often. Never
underestimate the power of just buying ram.

~~~
gnufx
> The obvious ones are fluid dynamics, weather simulations, n-body problems,
> and simulation of quantum systems.

Those are typically run on distributed memory; the question should probably
have been about shared memory. They're likely to be memory bound, depending on
the application, but it's not obvious that shared memory in ~100 cores would
win. Then consider checkpointing, for instance. (In my experience it's
bioscientists who think they need huge shared memory but may well not, or
couldn't feed it fast enough anyhow.)

Recent top supercomputers have had ~1PB or more total memory, though that's
pretty variable, and there probably won't be many jobs using a large fraction
of the machine. They may also (or only) have HBM.

------
chiph
> Most implementations of transactional memory are based on software. The
> POWER9 processor-based systems provide a hardware-based implementation of
> transactional memory that is more efficient than the software
> implementations and requires no interaction with the processor core,
> therefore enabling the system to operate at maximum performance.

Wow. You'd need a compiler that supports this feature (I think I read that GCC
4.7 has experimental support). This would support their traditional customer
base - banks and other firms that want atomic operations at speed. There's
probably some limitations around transactions across nodes, since you'd be
leaving the chassis.

------
ChuckMcM
Such an amazing machine. If I had a too much money and time on my hands I'd
put one of these systems into a "pocket" machine room (one cold, one hot
aisle). The ideas you could explore would be pretty amazing.

------
Koshkin
NetBSD wasn't mentioned so I am left wondering if it runs on this "toaster."

~~~
fredsted
Of course it runs NetBSD.

~~~
spijdar
Funnily enough, POWER is one of the notable platforms NetBSD has basically no
support for at all. NetBSD will run on 32 bit PowerPC Macs, but only those. No
support for 64 bit kernels or userlands, and only the most bare bones support
for early 32 bit OF power systems.

OpenBSD recently added support for the PowerNV/OpenPOWER platform, could
probably use their locore to bootstrap a NetBSD system. Still wouldn't run on
a PowerVM system like the E980 :)

------
frank2
Who is the typical customer for this product, and to what application is it
usually put?

~~~
imglorp
These guys are analyzing one use case where you can sidestep a clustered
Oracle pricing peak. If you can cram your whole thing in memory you have lower
latency from disk and also on slow remote DB queries and smaller Larry bill.

(zoom the pictures for details)

[https://precisionitinc.com/project/sample-total-cost-of-
owne...](https://precisionitinc.com/project/sample-total-cost-of-ownership-
tco-analysis-oracle-exadata-vs-ibm-power9/)

~~~
gigatexal
insane that oracle DB for that many cores costs 7M USD

