
The printer that wouldn't print: Fixing an IBM 1401 mainframe from the 1960s - rbanffy
http://www.righto.com/2018/09/the-printer-that-wouldnt-print-fixing.html
======
Animats
Fixing 1960s and 1970s electronics is hard. But they're fixing IBM's good
stuff, not junk.

It's rare to see a transistor opened up today to see what went wrong. From the
1950s to the 1980s, the USAF did that on occasion. The USAF had a project, the
Air Force Reliability Program, to make electronics more reliable. About 1% of
the electronics boxes in the USAF were stickered with a note like "USAF
Reliability Program - do not repair in the field, replace and ship back to
..." Back at Reliability Program HQ, technicians would figure out exactly what
had broken and document it. All the way down to the individual component level
and below. Transistors would be opened up and photographed under a microscope
to see exactly what went wrong. They found problems like this bad transistor
in the 1401, where a lead wasn't attached properly.

The USAF followed up by naming and shaming manufacturers. Microphotographs of
bad parts were published in Aviation Week. Parts got better.

~~~
de_Selby
I expected the debugging session to end when they identified the transistor
that was bad, but these guys were dedicated!

I've never seen a transistor can opened like that. I love how it's essentially
just a shrunken down version of the very first transistor ever made[1]

Great story about the USAF. It's a shame that circuitry is essentially a black
box nowadays.

1:
[https://physics.aps.org/assets/c5065b1e-4bf2-4501-a05b-6a841...](https://physics.aps.org/assets/c5065b1e-4bf2-4501-a05b-6a841be4559e/e16_1.jpg)

~~~
zenojevski
A transistor is just as much of a black box, only incredibly simpler.

You can decap hardware today in roughly the same way, and given of course
enough time, understand it, and eventually control it.

~~~
stephengillie
These days decapping involves sandpaper and a magnifying glass.

~~~
jacquesm
Most likely acid. Sandpaper will destroy whatever you are trying to decap if
you get close.

~~~
13of40
Used to be able to burn the package off with a torch and leave the chip
somewhat unscathed. I hate to think of the chemicals I was exposing myself to
when I did that as a kid, though.

------
scrooched_moose
I'm lucky if I follow 20% of these posts, but they are consistently my
favorite items and I look forward to each and every one. Similar to the
Dolphin Emulator blog, it's just a joy reading deep technical dives from
people who are passionate about their work.

Even not understanding most of it, I always manage to learn several
interesting tidbits.

------
Taniwha
Chain printers like this were amazing to watch, the chain had to move
amazingly fast and could break ... I worked with a Burroughs mainframe, we
were always cautioned never to stand either side of the printer, because
reputedly if the chain broke it could fly out the side and embed itself in the
wall (never actually happened to us)

On some printers you could change chains for particular print jobs - when we
printed some jobs we'd load up a chain with just digits on it (so the printer
could print an entire line while the chain moved 10 locations rather than 48
locations - theoretically 5 times faster.

Page layout was done with a loop of paper tape N positions long (that matched
the number of lines on a page) and 10 widem each one of 10 horizontal holes
corresponded to one of the fortran carriage control commands "1" meant "skip
until there's a hole in position 1 on the tape" \- you'd change to a special
loop when printing labels for example, or for different sized paper - if you
made a loop and were stupid enough not to have a hole in each horizontal
position at least once around a loop skipping to that position would result in
a "page throw" fanfold paper would exit the printer horizontally until it hit
the wall and would continue to do so until you ran out or someone noticed

------
inetsee
The very first programming job I had out of college was working on a DEC PDP-8
(not an 8-I or and 8-E, a plain 8), and I remember one time when it failed. I
ran a diagnostic program (from paper tape) and the result indicated a hardware
failure. We had a service contract, so we called DEC and they sent out a
service person. He re-ran the diagnostic and confirmed the fault. I expected
him to swap out the failed board, but no. He pulled the board out of the
chassis, un-soldered a transister, and soldered in a new one. I can remember
to this day, my amazement that he could repair a computer like that.

~~~
technofiend
I vividly remember walking into my high school computer lab to find our
PDP-11/34A partially disemboweled and someone hunched over the guts of the
machine. An update tape from Digital required wire wrap changes which were
included in the release notes.

------
drfuchs
Approximate translation of the "print storage" paragraph: "Fetching characters
to print via DMA consumed the entire memory bandwidth of the mainframe, so
printing caused all computation to stop; unless you leased the 132-word 'print
storage' hardware option for an extra $3000 per month (in 2018 dollars)." This
add-on looks to be about the size of a largish suitcase. One word in this
context was 12 bits, so a grand total of 1584 bits, plus lots of control
circuitry.

------
acomjean
Better than our office couldn't print story. The admin was in a car accident
in the late 1990s.

She was gone 3 days before we could no longer print. I remember talking to her
and her complaining that we were using too much space for emails, and whoever
set up the Network put the email and the print spooler on the same drive. I
told people if they cleaned out their inboxes printing should work again, and
it did. She returned a couple weeks later and fixed the problem in a more
permanent way.

------
joezydeco
I remember reading an old urban legend about someone that got the exact
sequence of characters on the print chain and wrote some code to print exactly
that line on the 1403.

The legend tells of the 1403 dutifully printing that line by firing all 132
hammers at once, causing the chain to violently break and whipsaw itself out
of the side of the printer cabinet.

~~~
vxNsr
> _You might expect that the 132 hammers align with 132 type slugs, so the
> matching hammers all fire at once, but that 's not what happens. Instead,
> the hammers and type slugs are spaced slightly differently, so only one
> hammer is aligned at a time, and a tiny movement of the chain lines up a
> different hammer and type slug. (Essentially they form a vernier.)
> Specifically, every 11.1 microseconds, the chain moves 0.001 inches. This
> causes a new hammer / type slug alignment. For mechanical reasons, every
> third hammer lines up in sequence (1, 4, 7, ...) until the end of the line
> is reached; this is called a "subscan" and takes 555 microseconds. Two more
> subscans give each hammer in the line an option to fire, forming a print
> scan of 1.665 milliseconds. 48 print scans give each hammer a chance to
> print each character, and then the 49th print scan is used for error
> checking. (For more details of this timing, see Manual of Instruction, page
> 37.)_

Based on this note at the bottom of the article that seems impossible, could
be I'm just misunderstanding what he's saying though.

~~~
kens
You are correct; it's impossible to fire all the hammers at once on the 1403
since they fire one at a time. Maybe the self-destructing printer is a
different line printer, or maybe firing the hammers in the right sequence
over-stresses the chain, but I suspect an urban legend.

------
vxNsr
Since I started college and really knew what breadboards were I always
believed they were only every used in schools or else for prototyping,
anything in production/commercial used etched silicon, that assumption was
killed today.

It now makes sense why it cost $20,000/month (today's $) to use this computer,
if you had to manually build each board, the cost for that alone is
astronomical...

I think I owe my ECE 101 professor an apology.

~~~
Animats
There are no "breadboards" in an IBM 1401. There's a wire-wrap backplane, but
that's a production technology. Even manual wire wrapping is very organized.
You have a list of X-Y coordinates to attach, and a supply of pre-cut and pre-
stripped wires of the right length. You put in all the shortest wires first,
then the next length, and so on. A wire-wrap gun-like tool does the twisting
and stops when the wire is tight. It's easy, and the result is reliable. Much
better than those solderless breadboards the Arduino crowd uses.

By the IBM 360 era, IBM had much of the production process automated. Wire-
wrap backplanes were wired by a Gardner-Denver CNC wire-wrapping machine.
IBM's "solid logic technology",a pre-IC technology, placed components in
little ceramic bases which were then canned. IBM pioneered automated weaving
of core memory. For 15 years, IBM was way ahead on automated electronics
manufacturing, and extremely profitable.

All this was a dead end. Making ICs by photolithography became cheaper in the
1970s.

~~~
kens
Good answer. A couple additional notes: the 1401's backplane was also wire-
wrapped by a Gardner-Denver machine controlled by punch cards. Field engineers
could modify the wiring with hand wire-wrap tools (to fix problems or add new
features), but most wiring was automated.

IBM was using CAD for their systems way back; the 1401's wiring diagrams were
called Automated Logic Diagrams because they were computer-generated,
originally using a vacuum tube IBM 704 or 705 computer.

I'm still not sure if IBM's solid logic technology (SLT) was a good idea, or
if they should have moved to integrated circuits much sooner.

------
chewyland
I find this stuff so incredibly fascinating that my hands are shaking...

------
Ricardus
If you like stuff like this, follow curiousmarc on youtube. He volunteers
there and helps with restorations, and does all kinds of cool restorations at
home, too.

------
dukoid
Does anybody have a scan of a sample printout? If I google it, I just get IBM
1403 results... :-/

~~~
dukoid
p.s. Looks like 1403 is the printer unit for the IBM 1401....

------
microtherion
Wonderful article, as always. It's impressive how detailed IBM's documentation
is.

~~~
rbanffy
What I thought was cool is that it was machine generated.

------
purplezooey
Get to work on the Holmes here with pliers and a blowtorch.

------
throwaway456321
I wonder how long it would have taken to debug this back in 1960

~~~
dsfyu404ed
Probably half the time or less since the people working on it would be more
familiar with debugging these sorts of systems.

~~~
kens
Yes, I expect field engineers back in the 1960s would have found the problem
much faster than we did. They had oscilloscopes and test equipment, so we
don't have much advantage there. They would be much more familiar with the
system and wouldn't need to keep looking stuff up like we did. Plus, they
would be motivated by anxious customers breathing down their necks demanding a
quick fix.

~~~
kw71
I think the digitizing oscilloscope is quite an advantage compared to the old
stuff ;)

------
jiveturkey
love that this is done using a garbage scope (rigol). even the garbage of
today is quite capable enough to debug the start of the art of yesteryear.

~~~
kens
I'm quite happy with the Rigol scope I used (DS1054Z), and would recommend it
to anyone who wants an inexpensive scope.

~~~
kw71
I am a die hard LeCroy user and fan, but recommend the rigol when asked what
to buy. For most people, my criticisms of it are of no concern (I am
frequently chaining instruments with GPIB and operating them remotely, and the
usb stuff on the rigol doesn't fit into that and just too slow.)

