
What Happens When You Mix Java with a 1960 IBM Mainframe - 3n7r0pY
http://thenewstack.io/happens-use-java-1960-ibm-mainframe/
======
Animats
I'm still struggling with my own legacy code problem. I'm reviving an old LISP
program from the early 1980s. Parts of it were written for the original
Stanford AI Lab SAIL system in the 1970s. It last ran under Franz LISP in
1986.

My current struggle is with one line of code:

    
    
       (defun getenode (l) (cadr l))
    

That ought to be simple enough. But it's being applied not to a list, but a
"hunk". A "hunk" is an obsolete MacLISP concept.[1]. It's a block of memory
which has N contiguous LISP cells, each with two pointers. This is the memory
object underlying structures and arrays in MacLISP. Macros were used to create
the illusion of structure data objects, with hunks underneath. However, you
could still access a "hunk" with car, cdr, cxr, etc.

I'm converting this to Common LISP, which has real structures, but not hunks.
That, with some new macro support, works for the regular structure operations.
So far, so good.

But which element of the structure does (cadr l), which usually means the same
thing as "(car (cdr l))", access? (cadr (list 0 1 2 4)) returns 1, so you'd
think it would be field 1 of the structure. But no. It's more complicated and
depends on how hunks are laid out in memory.

The Franz LISP manual from 1983 [2] says _" Although hunks are not list cells,
you can still access the first two hunk elements with cdr and car and you can
access any hunk element with cxr†."_ At footnote "†", _" In a hunk, the
function cdr references the first element and car the second."_ This is
backwards from the way lists behave.

A blog posting from 2008 about MacLISP says _" A Maclisp hunk was a structure
like a cons cell that could hold an arbitrary number of pointers, up to total
of 512. Each of these slots in a hunk was referred to as a numbered cxr, with
a numbering scheme that went like this: ( cxr-1 cxr-2 cxr-3 ... cxr-n cxr-0 ).
No matter how many slots were in the hunk, car was equivalent to (cxr 1 hunk)
and cdr was equivalent to (cxr 0 hunk)."_ Note that element 0 is at the end,
which is even stranger. The documentation is silent about what "cadr" would
do. Does it get element 2, or get element 0 and then apply "car" to it?

The original code [3] contains no relevant comments. I'm trying to figure out
from the context what the original author, Greg Nelson, had in mind. He died
in 2015.[4]

[1] [http://www.mschaef.com/blog/tech/lisp/car-
cdr.html](http://www.mschaef.com/blog/tech/lisp/car-cdr.html) [2]
[http://www.softwarepreservation.org/projects/LISP/franz/Fran...](http://www.softwarepreservation.org/projects/LISP/franz/Franz_Lisp_July_1983.pdf)
[3] [https://github.com/John-
Nagle/pasv/blob/master/src/CPC4/z.li...](https://github.com/John-
Nagle/pasv/blob/master/src/CPC4/z.lisp) [4]
[https://en.wikipedia.org/wiki/Greg_Nelson_(computer_scientis...](https://en.wikipedia.org/wiki/Greg_Nelson_\(computer_scientist\))

~~~
sillysaurus3
There are only a few possibilities for what (cadr hunk) could mean. One way to
solve this is to try them all and see which one runs successfully.

Based on †, it sounds like (cadr (hunk (hunk 1 2) 3)) should return 2.

Is the old LISP code available online somewhere? I'm curious to see it.

~~~
Animats
See the link listed above: [https://github.com/John-
Nagle/pasv/blob/master/src/CPC4/z.li...](https://github.com/John-
Nagle/pasv/blob/master/src/CPC4/z.lisp)

I put all the code on Github. The oldest version of each file is exactly what
ran in 1986.

The code is delicate. It's a theorem prover, and there's much manipulation of
complex data structures, with little explanation of what's going on. The
overall theory is documented; this is the original Oppen-Nelson simplifier and
there are published papers. But the code has few comments.

~~~
sillysaurus3
getenode is only called in a few places, and each call looks like:

    
    
      (getenode znode)
    

i.e. getenode is only called on znodes. The definition of makeznode is:

    
    
      (defun makeznode (node)
             (prog (l znode)
                   (setq l (list (list '(1 . 1) node)))
                   (xzfield node (list l node nil))
                   (setq znode (tellz l node))
                   (or (null znode) (eq znode node) (break makeznode))
                   (return l)))
    

So it seems like getenode is called on a regular list whose structure looks
like

    
    
      (list (list '(1 . 1) node))
    

Assuming "node" is an enode, the way to access it would be (cadar znode), not
(cdar znode). Try changing the definition of getenode to

    
    
      (defun getenode (l) (cadar l))
    

and see if it runs.

~~~
Animats
I thought that way at first. But the program breaks when (getenode znode) is
called on an item of type "node", not a list. This happens when (interntimes)
takes the path which ends

    
    
        (or (setq znode (tellz l node)) (return t))
        (or (eq node (getenode znode)) (zmerge node (getenode znode)))))
    

and (tellz) has taken the path which ends (return (baserowz* i)). "baserow*"
is an array which contains links to "node" items, not list cells. What
"z.lisp" and "ze.lisp" are doing, by the way, is solving systems of linear
inequalities by using linear programming on a sparse matrix.

Also, see

    
    
        (defun isznode (x) (and (hunkp x) (= (hunksize x) 8))) ; original
    
        (defun isznode (x) (equal (type-of x) 'node))   ; CL version
    

which indicate that znodes are hunks/structures, not lists. This is
inconsistent, yet somehow it used to work.

(If you want to talk privately about this, I'm at "nagle@animats.com". Too
much detail for HN.)

~~~
kazinator
or abuse, ouch!

    
    
       (unless (setq znode (tellz l node))
         (return t))

------
skissane
Is someone in the US government really still using an IBM 7074? Really? I'm
shocked. How could that possibly be cost-effective?

Actually, this blog post makes the story clearer:

[http://nikhilism.com/post/2016/systems-we-
love/](http://nikhilism.com/post/2016/systems-we-love/)

It isn't a physical IBM 7074.

When it came time to migrate from 7074 to S/360, rather than rewriting their
7074 software, they just wrote a 7074 emulator for S/360\. And, it sounds
like, they are still running their 7074 software, under their 7074 emulator,
most likely on a recent z/Architecture mainframe.

The article makes it sound like people still use "1960s mainframes" when I
very much doubt anyone is still running 1960s hardware in production. People
use modern machines–modern IBM mainframes, which are multicore 64-bit
processors–or other mainframe vendors such as Unisys or Fujitsu use mainly I
believe x86-64 running Linux running a software emulator for the old mainframe
CPU.

A lot of legacy, sure, but I think this article makes it sound even more
legacy than it really is.

~~~
YeGoblynQueenne
It's possible Ms Belotti and her team didn't realise they were working with an
emulator. Mainframes being what they are, the programmers were probably never
in the same room as the machines they were progamming, and it does take a bit
of digging to figure out that the architecture you see before your eyes is
emulated on another machine (like in "The Story of Mel").

Still, even the blog post you link to doesn't make it absolutely clear that
java was running on an emulated s/370\. It says that the decision was made to
emulate the older architecutre rather than rewrite the old programs, but then
it goes on to say "These are still operational". Does it mean the old
programs? Or the old machines? It's hard to say.

As to how unlikely it is to see a very old machine still in use, instead of
one made in more recent times, last year I talked to an engineer who claimed
he had seen a PDP still in operation in some transport company if memory
serves.

~~~
abakker
I believe (a coworker of mine worked on S/360) that all the old software can
be effectively run through emulators on the current system z. The feeling was
that once you had done the development and testing of an older system, IBM was
never going to force you to rewrite your code. As a result, the upgrades were
pretty seamless over the years. Many of those customers never had to upgrade
and never had a reason to.

------
mbellotti
This article was such a pleasant surprise! I loved talking at Systems We Love
and I love talking about legacy architecture in general. Bryan and the Joyent
team were very accommodating and understanding. The White House is an amazing
place to work (even now), but it's not a system that understands conference
talks very well. Wish we could have a video, but it was a heavy lift just
getting the bureaucracy okay with the idea that I was going to talk about
information that was already public, without naming agencies or projects and
without going into detail beyond what was in a user manual and that this was
not a security risk.

~~~
benballjr
Hi Marianne! I'm with The New Stack (not the writer of this article), and was
wondering about the video myself. Interesting to see you address that there's
definitely no public facing version, as it seems a lot of the commenters here
would love to see it. I can only imagine the bureaucrat complexity involved.

~~~
mbellotti
Oh yeah, when I talked to the agency that owns the 7074 (more on this in a
second) they were like "You can't do this because we will get hacked"

"How are you going to get hacked if I talk about your mainframe? It's not
connected to the public internet, is it?"

"No. Well... we don't know... but ... hackers! Hackers are really smart
Marianne."

Part of the compromise was that I promised I would only use information that
was already available publicly through government reports and news articles. I
went back through my talk and documented where each fact was already published
somewhere else until they were comfortable with it. So the ambiguity on
whether the 7074 was the actual machine or an emulator was deliberate... there
were certain things I could not find a public comment on and therefore agreed
to avoid making direct statements about.

This all seems super annoying, but it makes sense when you realize how heavily
scrutinized public servants are. In the end they are only trying to protect
me, my organization and Obama's legacy. Three things that are really important
to me. So I can't exactly blame them for it. I was happy to be able to find a
middle ground where they felt comfortable, the organizers weren't too badly
inconvenienced and I got to give the talk I wanted to.

------
YeGoblynQueenne
>> It was at this point that the seasoned data architects in the department
began expressing their exasperation. “15 years ago, everybody was telling us
‘Get off the mainframe, get on AT&T applications, build these thick clients.
Mainframes are out.’ And now thick clients are out, and everybody’s moving to
APIs and microservices, which basically are very similar to the thin client
that a terminal uses to interact with a mainframe.”

A.k.a. "Them as not knows their history are doomed to repeat it". But I
suppose it's more the case of business imperatives than real ignorance that
drive this mad race to make new stuff that works worse than the old stuff only
so we can then go back to the old stuff with a different name.

Btw, that lady is my new tech hero:

 _“The systems that I love are really the systems that other engineers hate,”
Bellotti told the audience — “the messy, archaic, half chewing gum and duct
tape systems that are sort of patched together._

<3 <3 <3

~~~
eternalban
"15 years ago" doesn't sound right.

Thick clients were being pushed in mid 90s. But then Clipper Chip plans went
south ...

~~~
toyg
Consider that state/federal bureaucracies get on "trends" with a delay of 2 or
3 years minimum, and the story is probably from 2015 or thereabout - the
agency was launched in 2014.

------
jonnycowboy
The thing is that there was a lot about those old systems that was slow, so
you were very, very careful how you programmed. You tended not to use vast
library stacks, you went close to the metal and you coded in languages like
Assembler, COBOL or FORTRAN. I/O was often run through specialised co-
processors (such as IBM's channel processors) and the terminals could
sometimes help too.

I have friends who have been looking after legacy applications for an airline
running on Unisys. The core apps for reservation, Cargo booking and
weight/balance were written in FORTRAN. In recent times, the front end was
written in Java to give web access. They tried to rewrite the core apps but it
was impossible to do so and get the performance.

~~~
YeGoblynQueenne
>> They tried to rewrite the core apps but it was impossible to do so and get
the performance.

Well, Cobol is a bit like the C of mainframes - you can manipulate memory
directly and so on. You can't really do that sort of thing with Java.

~~~
stefs
a) if it was really running on the old hardware; in that case ruby on a modern
machine would have been several magnitudes faster than the original code - at
least because of the faster IO

b) if the whole thing was indeed running in an emulator, the emulation
overhead would have negated all direct memory access advantages

~~~
wahern
Emulators on mainframes are much more sophisticated and performant than is
typical on x86 and ARM platforms. The hardware and even software is often
designed with emulation in mind, not just for backwards compatibility but for
forwards compatibility, too.

~~~
andai
Do you mean to say that they were running a mainframe emulator on a mainframe?

~~~
wahern
Compilers for some IBM mainframes have for decades (since the 1970s, I think)
targeted an intermediate virtual machine instruction set which is then
translated on-the-fly to the local architecture by the OS upon execution. So
in the case of IBM their machines are truly built with both forward and
backward compatibility in mind. The pointers for this instruction set have
been 128 bits since the beginning, long before 64-bit hardware even came into
being.

Some (or all?) of the latest Unisys mainframes run on Intel Xeons, but with
custom chips for translating the machine code of their old architectures.

I don't work in this area. I just like reading about it. Though,
unfortunately, it's difficult to find clear specifications and descriptions on
how these architectures work.

For example, Unisys' 2200 ClearPath architecture is one of the (if not _the_)
last architectures still sold that uses signed-magnitude representation, as
well as having an odd-sized integer width of 39 bits. (INT_MAX is 549755813887
and INT_MIN is -549755813887, and the compiler has to "emulate" the modulo
arithmetic semantics required of unsigned types in C. ClearPath MCP is also a
POSIX system, and has to emulate an 8-bit char type.) I discovered that you
could download the specification for their C compiler for free online, which
was useful when discussing the relevancy of undefined behavior in C. But AFAIU
(and this is where finding concrete details is more difficult) the latest
models of the ClearPath line use Xeons with custom chips bolted on to help run
the machine code of the older architecture. In any event, the point is that
while the old architecture is arguably emulated, it's not a pure software
emulation that you might assume, and the resulting performance is better than
the previous models of those mainframes, which were still being built at least
until a few years ago. In other words, direct memory access isn't ruled out
because the I/O systems may have been intentionally designed to work
efficiently in a backwards compatible manner.

~~~
dfox
The 128bit pointer intermediate code is used on what IBM calls "midrange
systems" (ie. AS/400), not mainframes. IBM mainframes execute their machine
code directly and the ISA is designed such that it allows for efficient
virtualization since beginning and is extensible and backwards-compatible.
Otherwise the IBM mainframe magic is in IO offload and truly immense memory
bandwidth. On the other hand, Unisys systems use architecture that is
significantly different from what today's programmer would expect, with
completely different memory model originally implemented in hardware (which
essentially combines the memory protection model implemented on AS/400 in
software with lisp machine-style pointer tagging).

------
nl
Did anyone read this and go "hu"?

There's this whole thing about how they are getting data from mainframes where
"the data was being returned in between one and six milliseconds".

But then: "harvest that data from the magnetic tape and load them up into more
traditional databases. That Java application was extracting the data from the
databases"

But _then_ : "But the data from the mainframes was actually arriving (from its
new home in the database) in less than six milliseconds. The bottleneck was —
of course — the Java application."

So of course it is entirely possibly to write slow Java applications. But then
the story seems to end! So what happened? Did they fix the application?

------
geodel
I have heard from many of my friends about massive projects of 'modernizing'
mainframe applications with Java stack. Java did not deliver improvement that
management was expecting. Once the project consumed all the budget for ~25%
completion, they were scrapped.

I think, though without any proof, that overreliance of 100s of mixed quality
libraries, combined with 'best practices' of enterprise development and heavy
application of design patterns creates a very large surface area for change.
This makes reasonable translation of functionality to Java almost impossible.

~~~
neeleshs
Was it Java that did not deliver improvement, or was it the team? :)

~~~
geodel
I am inclined to say Java as I have not seen mythical teams who work on Java
without whole caboodle of 'Enterprise Apps' culture.

~~~
electrum
You can write modern software on Java without using "enterprise" features:
[https://github.com/prestodb/presto](https://github.com/prestodb/presto)

~~~
geodel
Nice. It may be the best way to organize project this large. I know this is
maven thing and once using everyone's favorite Intellij IDEA it should not
matter but I personally find ~1000 directories for ~4000 files a kind of
Javasim.

------
walrus01
When I first saw the Chromebook, my thought was that Larry Ellison's 1998
dream of a network computer thin terminal had come true. It's smarter than a
true thin terminal, but everything lives in the cloud (err, butt).

~~~
bitwize
That Chrome extension is clbuttic.

~~~
TeMPOraL
Indeed.

> _but everything lives in my butt (err, butt)._

I had to open the comment in porn (er, incognito) mode in order to parse it
:D.

------
mtdewcmu
"It used magnetic-core memory (instead of cathode ray tubes) according to
Bellotti"

I didn't think magnetic-core memory and CRTs were interchangeable...

~~~
dpcx
I think the implication wasn't that they were interchangeable, but that the
machine they were expecting to see used CRTs, and instead, they found it using
MCM.

~~~
mtdewcmu
I didn't think that CRTs were a memory technology, but apparently they have
been:
[https://en.wikipedia.org/wiki/Williams_tube](https://en.wikipedia.org/wiki/Williams_tube)

------
brokentone
I'm not sure the article did well answering the headline. I now know that
there is a mainframe (emulator it seems from additional research) that returns
API responses very quickly (1-6ms) TO a Java app... which apparently is
inefficient as the page render takes 6-10 seconds.

Interesting problems not terribly well described in the post, would love to
read more.

~~~
robertlagrant
Yeah it describes pretty basic "discoveries", and reads a bit like a book
report. Would love some detail.

------
wolf550e
An explanation from 'bcantrill about why the talk was not recorded:
[https://lobste.rs/s/q2xz14/systems_we_love_videos/comments/i...](https://lobste.rs/s/q2xz14/systems_we_love_videos/comments/if9yox#c_if9yox)

------
mleonhard
She likes those crufty legacy systems only because she works with each one for
a short time. If she had to work on one every day for years, she would hate
it, too.

------
stevehiehn
Hmm, interesting. My bet is that the Java devs were not great at implementing
concurrency. But who knows.

~~~
kevin_thibedeau
Probably lots of XML transforms going on to be enterprisey.

------
altitudinous
If it ain't broken, don't fix it.

------
chris_wot
You get sued for violating copyright?

------
platz
Remember COBOL.net !

------
asjo
EBCDIC stacktraces!

