
Powering a Mainframe - jandeboevrie
https://blog.mainframe.dev/2019/07/powering-mainframe.html
======
abraae
I hate big power. When I was an IBM engineer, power into the computer room was
a big deal. Big diesel generators feeding into the main switchboard along with
redundant mains connections.

One incident I remember is when an electrician was working on one of the power
feeds while it was live. Foolish, but the kind of thing people did when
getting approval to shut off power was a bureaucratic nightmare.

The sparky was tightening down a bolt on one of the phases with a big crescent
spanner when he inadvertantly made contact with another phase.

There was a flash that lit up the room and a loud bang. His crescent spanner
literally melted in the middle. Incredibly he was uninjured, although he did
take the whole room down, which was an extremely serious matter.

Modern computers that plug into the wall socket are much more pleasant to work
on.

~~~
ChuckMcM
Nice, my favorite big power story was a technician who was bringing up the
next set of "rows" in a very large web scale data center.

One component of that task was running thousands of feet of CAT-6e cable. Of
course you want to be sure your spool of CAT-6e cable has no wire breaks in it
before you start using it[1] to wire up racks so there was a step that
involved putting a connector on the stub of cable that was inside the reel,
and a connector on the end of the cable that came off the reel, and then
hooking up an instrument between them to measure the
resistance/propagation/etc.

It is convenient to work at waist height for this, so our hapless technician
had the cable spool put on a convenient big square brick like metal structure
outside the building. He added the connectors, hooked up the tester, and the
tester exploded. (well not completely but it did experience some rapid
unplanned disassembly due to a large number of capacitors suddenly not
existing.

The "brick like structure" that the technician was using as a table top
happened to be one of the transformers that was converting high voltage down
to 440V 3-phase for the data center. When he effectively shorted the cable
loop with his tester he air coupled into the field around the transformer and
put > 400V into the tester. Whoops!

[1] tracking down a bad network cable is not a lot of fun.

~~~
emeraldd
With that much power floating around, I would have expected the transformer to
have better shielding/warnings. Or is that not practical?

~~~
ChuckMcM
It had the required shielding and warnings about opening the box. That said,
the spool of cable was likely 2500' of CAT-6e cable. In the root cause
analysis it was estimated the voltage potential on the cable was between 210
and 280V and the current levels could have reached 500mA (a bit more than 100W
of 'leakage'). That was well outside the what the test gear was "expecting"
from a cable that it was testing.

------
infosecfriends
If anyone is in Victoria, Australia and wants to play with an IBM z9
mainframe, i picked one up for about $500 and although I've managed to power
and boot it, i haven't managed to load zOS on it properly. Got some time to
spend on it and would love it if anyone else is interested in
experimenting/getting it up and running. I also bought the relevant storage
array (but no disks). Unfortunately that takes up a heap of space and I'm
looking to offload it, it's free if anyone wants it, otherwise ill probably
have to scrap it :(

I usually lurk here but feel free to reach out via twitter (same username)

------
derefr
I’ve seen those “support element” laptops hanging off of racks before, but I
always assumed that in this case (IBM mainframe, IBM support element, vertical
integration) they were not laptops as such, but rather thin clients in the
most literal sense: just IO interfaces with a ribbon cable feeding back to the
mainframe, where the framebuffer of the mainframe’s dom0/maintenance VM was
pushed over VGA and keyboard+mouse+peripherals were returned over USB.
Essentially as if the support element were a Thunderbolt Display with USB HID
devices plugged into it. Like a modern VT100!

I’m honestly surprised to learn that I’m wrong, as having the laptop be a real
laptop would introduce all sorts of headaches compared to having the laptop be
a dumb terminal—it’d have active components like disks and fans that could to
burn out (and not be hot-swappable like the server’s own components are); and
more importantly, it’d have to talk to the server using some form of network
interface (unless it’s just a glorified glass terminal for a USB-serial
adapter.) In which case, bad behaviour on the server—exactly the case where
you need a local admin console—could block or steal resources from your
connection attempt. You may as well just be SSHing into the dom0 from across
the room over the SBC VLAN at that point.

~~~
dfox
Support element has to be more or less full-blown computer as the mainframe
itself does not have any real firmware to boot^H^H^H^HIPL itself. Before
support elements the IPL process involved initalizing various things by
toggling switches on the console.

Also why the mainframe would even have a framebuffer?

Providing a console interface (ie. 3270 terminal emulator) to whatever OS runs
on the mainframe is only small part of what SE does.

~~~
throw0101a
> _real firmware to boot^H^H^H^HIPL itself._

Have you considered using ^W?

~~~
hackmiester
Just a word of caution for anyone reading this. I highly discourage anyone
from actually learning ^w. I learned it, then had to forcefully un-learn it
when I kept accidentally closing tabs on my browser, other people's browsers,
my file manager, etc. etc. ad nauseam. Don't make the same mistake I did, just
press backspace a lot. I know life is short but the sudden rage of having
unexpectedly closed a tab is just not worth the added efficiency. I am
passionate about this subject; however, I mean no disrespect for those who
have (somehow) managed to use ^w productively.

------
rvz
I like seeing posts like this showing that there is still interest in
operating with mainframes and bare-metal servers these days.

My biggest fear is that you are in the minority of developers that can single-
handedly operate one in the age of buzzword-ridden lingo such as AWS, GCP,
Heroku, etc which forces a dogma to run rampant in our industry to host all of
your startup/company only on other peoples's servers rather than to setup up
your own in-house bare-metal servers instead.

So I am very impressed to see this, as an added bonus it is a mainframe.
Please post more of this.

~~~
vbezhenar
It's not about clouds. The problem is with mainframe manufacturers. Their
prices are ridiculous, so no sane man with limited budgets would buy it. I can
build incredible powerful server from a desktop or workstation components for
a few thousand dollars. Probably can reduce it to hundreds with using old
parts. I can increase price to x2 and buy some HP or Dell blade and that would
be real server hardware. But nobody is going to increase price to x100 and buy
mainframe. That's just too much.

~~~
crb002
IBM is dead until it gets a mainframe in AWS region data centers. Mainframe is
still king of CA in CAP.

~~~
znpy
Last time I checked IBM was alive and doing well.

~~~
jdsully
They’ve been in a 10+ year turnaround. “Doing well” is debatable.

------
idlewords
This is a fascinating bit of nerdery. Would someone be willing to explain what
mainframes are used for in a modern context, and how they compare to a really
high-capacity x86 machine? I know that one difference is that you can hot-swap
pretty much any component on a mainframe, but would be very curious to learn
more about these beasts.

~~~
derefr
A “really high-capacity x86 machine” is really just a machine with a lot of
CPUs and memory. It _is not_ a machine that can feed each one of those CPUs,
or DMA to each one of those memories, from separate network or disk IO
sources, all at once. Basically, there’s only one PCI-e bus, and it’s a
bottleneck. That’s why, for IO-intensive operations like search indexing, you
don’t scale x86 vertically, but rather horizontally, with map-reduce
strategies: you need each core and each DIMM of memory to have a commensurate
IO bandwidth available to it. And that’s also why a 64-vCPU VM on shared
hosting will always underperform a 64-core dedicated instance from the same
provider: with the dedicated single-tenant host machine, you’re not fighting
over an “oversubscribed” PCI-e bus.

Mainframes are built differently, such that each CPU and NUMA zone of memory
has its own (virtual) private bus to its own (dynamically) dedicated IO
peripherals—it’s like a little circuit-switched network with CPUs “placing
calls” to peripherals. And, because of this, mainframes _do_ let you just
scale your problem vertically, without IO bandwidth limitations ever getting
in the way. They function less like like one computer, and more like a Mosix
cluster that all happens to be running on a common backplane. But, unlike
Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and
the whole thing provides hotswap robustness against itself rather than that
robustness being partitioned per machine.

~~~
rbanffy
The other day I described a modern mainframe as something that resembles a
cluster where CPUs are connected through very high-speed low-latency buses (to
the point they can behave like a single-image NUMA machine or be partitioned
into multiple smaller ones) connected to multiple networks of specialized
computers each running their own tiny specialized OSs that manage IO over
different buses. It's common for them to have hundreds of PCI-e buses (and
cards connected to storage, communications, other mainframes - for failover or
clustering - etc).

An x86 server is a computer built around its CPU. A mainframe is a computer
built around its buses.

~~~
0815test
"Single system image" is a real approach to clustering, if a bit of an
experimental one. There used to be a variety of toolsets attempting to provide
this functionality under Linux (e.g. OpenMosix, OpenSSI) but they all seem to
have bitrotted by now.

~~~
rbanffy
I almost forgot about those. I'll have to re-read a couple papers.

------
Lowkeyloki
I looked at the author's previous post where he details the process of buying
the thing. A fair amount of it goes over my head, I must admit. But that final
price tag.... You've gotta really love mainframes. I can't imagine the utility
bills for it. The power usage was _reduced_ to 1.7kW. I guess some people buy
a sports car and some buy a mainframe. Personally, I'd never be able to drop
that kind of money for a hobby. I think emulating a mainframe on a fancy PC
would scratch my itch. But to each their own.

~~~
titanomachy
If you asked me how much I thought 700kg of computer should cost, I probably
would have guessed more than $12k. Is this thing modern? Specs don't seem too
impressive but I don't know anything about these things.

~~~
andrewcchen
In a previous post he said it's 8 years old

~~~
yjftsjthsd-h
I was about to think that was old, but then I remembered that mainframes, as I
understand it, can aim for decade+ uptime, so that might actually be
reasonably fresh?

------
C1sc0cat
Interesting it runs straight off normal 3phase mains - the only mainframe I
worked (a bit ) on was a small Unisys one and it had its own 400v supply as
part of the set up.

~~~
blattimwind
400 V is normal 3 phase mains in about 85 % of the world, but whether
consumers have 3 phases available or get 1 or 2 phases varies.

~~~
C1sc0cat
Ah yes I should have been specific I should have said a 400v DC.

------
nimbius
I was a former mechanical maintenance tech for an auto manufacturer, so this
story feels familiar to connecting a CNC lathe, or a multi-axis mill, but
reading this guys ham-fisted accounts made me pretty livid..

>The datacenter staff, not needing a forklift in their day-to-day, had managed
to solicit the services of a forklift, the associated operator, and some very
handy folks to help navigate the mainframe from the storage space to its final
location.

Since this thing can approach the cost of a rolls-royce,and weighs half as
much as a prius, movers and millwrights should have been called in. watching
two chubby guys in tennis shoes and no gloves move a 700 kilo server in
reverse would have given me a heart attack.

> The various guides did hint towards how the phases are connected to various
> Bulk Power Regulators (BPR) but nothing was very definite.

that should have been your clue to hire an electrician. there is no such thing
as a bulk power regulator. this is a BUCK POWER REGULATOR.

>I tried to do my best to construct a power cable

Make sure you tell your building supervisor and the fire brigade you "tried
your best." there is no "tried your best" section in the electrical code. You
either construct a cable to spec, or not.

>Now, I assumed that given there is no neutral it has to be a delta system

Since youre handling voltages that can kill you, perhaps a meter?

>The uncertainty was which phase-pair powered the lowest BPR? I guessed A-B

OR your BPR explodes, your HV breaker trips, and you wipe out power for half
the building. Christ on his throne.

~~~
racingmars
> there is no such thing as a bulk power regulator. this is a BUCK POWER
> REGULATOR

No, in IBM mainframes, the rack component that takes the mains AC and starts
doing something with it is, indeed, called a "bulk power regulator". For
example, IBM part number 6186-7040, "Bulk Power Regulator".

> that should have been your clue to hire an electrician

Note in the article he did say "I confirmed the wiring scheme with some
friends I have that do the power installations at Dreamhack, and even had them
do the wiring as they have all the qualifications needed to do so safely and
legally."

------
6thaccount2
I'd love to play with one as well and look forward to the later posts in this
series.

There has been a lot of mainframe related posts on HN lately.

~~~
tyingq
You could download the Hercules emulator and either a very old legal copy of
MVS, or a not-legal torrent of z/OS.

------
dsfyu404ed
TL;DR put it where you want it and plug it into a 3-phase outlet.*

*which slightly more complicated in Europe.

~~~
magduf
Why? It shouldn't be. They use 3-phase in Europe just like in the US, for
industrial locations. The differences in residential or lower-voltage
standards really have nothing to do with industrial power availability.

~~~
dsfyu404ed
The author described some shenanigans he had to go through when wiring the
power cord to get the power supplies (which expected voltage that is typical
in the US) to run right. In the US it would have been a simple matter of using
a power cord in the standard configuration and plugging it into the wall.

~~~
magduf
Looking closely at the photo of the power supply in the article, it looks like
it's quite flexible and can handle different voltages, as long as they're
3-phase. It even lists different amp draws for the different voltage levels.
In addition, it looks like it can handle 380-570V DC. With today's switching
power supply technology, it's not hard to make them able to accept widely
differing input voltages, so this is commonly done so you only need one part
to cover the entire worldwide market. It looks to me like you can just plug
this thing into any 3-phase power you want really.

As for a "standard configuration" in the US, there really isn't such a thing
for 3-phase power. There are a few NEMA standards used in industrial
applications, but a lot of times, things like that are hard-wired. This isn't
a home appliance.

