

Liquid-Cooled Supercomputers, to Trim the Power Bill - digital55
http://www.nytimes.com/2014/02/12/business/international/improving-energy-efficiency-in-supercomputers.html?ref=science

======
josh-wrale
In my experience, RAM needs to be replaced VERY OFTEN in HPCs. I seem to
remember reading somewhere that motherboards that go into a mineral oil bath
are basically impossible to clean afterwards. This point seems important,
since mineral oil is used here for its electrically insulating properties.
There are many contact pads in the marrying of RAM DIMM to slot. Ergo,
replacing sticks of RAM in such a supercomputer would probably be a nightmare.
And this nightmare would likely be frequent in its occurrence.

I also take issue with the first sentence of the article. Submerging a home
computer in mineral oil is a common practice. Their "technology" editor needs
some more technology chops, it would seem.

Another note: Let's hope they aren't secretly using polychlorinated biphenyls
(PCBs) in place of mineral oil to give themselves a boost. (see Wikipedia)

~~~
bashinator
I've read a similar article sometime last year, about Intel testing immersion-
cooled blades. That solution would make more sense IMO - blades are already
designed to be a single FRU, so if any component were to die, you'd just pull
the blade from its chassis. I imagine the process of safely disconnecting the
fluid hookups could be integrated into the blade chassis packaging system.

------
linksnapzz
Not just "one" model of Cray-at least the Cray-1, XMP, C90, T916/932, T3E…were
cooled by liquid HFCs.

At five million+ $$ per system, the cost of using Fluorinert as opposed to
mineral oil in the coolant cycle isn't even a rounding error.

If you spill mineral oil while swapping internal components, it's a trip &
fire hazard. HFCs evaporate quickly, and aren't flammable.

~~~
VLM
You might be surprised at flourinert cost, when that was contemporary rather
than ancient history I was looking into FC-40 and the like for some ham radio
gear cooling (hey, why not?). Believe it or not, FC-40 retails today for about
a kilobuck per 2-liter bottle. No I'm not kidding. So a couple thousand liter
tank is about a megabuck. So its not a rounding error.

~~~
qbrass
You could have picked a Cray T94 full of it when it was for sale on Ebay.

[https://news.ycombinator.com/item?id=4560173](https://news.ycombinator.com/item?id=4560173)

------
VLM
Coming from a EE background, sounds like total bogosity relying on the
customers not knowing anything about electronics. Magically dripping oil on a
piece of electronics makes it use half the power, yeah get back to me on that.

I never knew that adding massive capacitance to high speed signalling lines
made their drivers draw less current (super sarcastic here, its the other way
around)

I've dunked my share of power resistors, precision resistors, and the
occasional semiconductor in oils of various sorts and nothing too exciting
happens. Some electrolytic caps are not very well sealed and leak oil in and
(conductive) electrolyte out. Some component marking inks are oil soluble and
some inks are semi-conductive, which can be a problem.

~~~
mscman
It's not that the components themselves draw half the power, it's that you
don't need to provide as much power to cool them. There's a typical rule of
thumb in datacenter planning that for every 1W of compute power, you need 1W
to cool it. If you can cut the cooling power, you can cut almost 50% of your
datacenter cost.

Obviously you won't eliminate cooling cost entirely, and it's quite possible
the overhead of replacing oil and extended maintenance times could offset
these power savings.

~~~
welterde
Why not go for warm water cooling then (like SuperMUC[1] for example)?

Should get you the same benefits as submerging it in oil without most of the
drawbacks.

[1]
[http://www.lrz.de/services/compute/supermuc/systemdescriptio...](http://www.lrz.de/services/compute/supermuc/systemdescription/)

