
Our Government Runs on a 60-Year-Old Coding Language, and Now Its Falling Apart - gHeadphone
https://onezero.medium.com/our-government-runs-on-a-60-year-old-coding-language-and-now-its-falling-apart-61ec0bc8e121
======
theincredulousk
COBOL isn’t falling apart, the systems are exceeding their design limits. “Oh
if it were just written in a modern language everything would be fine”. Not
true - we create modern systems every day that would choke and die if they had
to scale several orders of magnitude instantly - in fact it happens about
weekly when an unsuspecting link hits the HN front page. Also, I would wager
90%+ of software written today is 10-100x more fragile over time than whatever
COBOL software is on these machines. What do you think the chances are of
those 100 fancifully orchestrated micro-services running 10 abstraction layers
up on 3rd party infrastructure are of running at all a decade from now, never
mind within their original design constraints.

Old doesn’t necessarily mean worse and newer doesn’t necessarily mean better,
but that logical foundation is easy to forget in a tech culture so often
myopically focused on innovation(tm) and short-term profit or austerity above
robustness.

The relative rarity of COBOL programmers aside, this is a just an garden
variety f* up of keeping legacy systems around too long because they aren’t
“broken”. And like technical debt, the money and effort to fix them, despite
many warnings and pleas, is called a waste because there is no immediate
benefit or profit.

Then something like this happens, and everyone forgets the shortsighted
management or decision makers that ignored the risk and collected bonuses and
praise for saving the Corp/taxpayers money. Those who warned about the debt
coming due get nothing except being able to say “I told you so” for a few
weeks until the whole thing blows over.

~~~
gregjor
I agree with what you wrote, this is not a problem with the COBOL language,
except in the narrow sense that it’s not taught anymore and probably 95% of
programmers working today have never seen COBOL. A professional programmer
with some experience can learn COBOL in a few weeks. It’s not Linear A.

The cost of fixing technical debt has to outweigh the cost and risks of
replacing legacy systems. Most ground-up rewrites fail, and that’s even more
of a problem in the government sector. A working system with problems is still
better than an imaginary system that will probably never go live. Google IRS,
California DMV, Oregon utilities, etc. etc. for examples of how these rewrites
actually work out.

~~~
theincredulousk
Yes I do remember reading the one - maybe it was Google IRS - that they tried
replacing cobol with Java.

Maybe in some cases they just need a new IBM mainframe instead of re-written
software.

~~~
thu2111
Emulating mainframes isn't an uncommon approach.

But, switching to something like Java would be far better. COBOL on mainframes
has nothing to recommend it. There are good reasons nobody chooses to write
new systems that way. For one, it's a single vendor tech in which that vendor
is a huge firm in terminal decline and thus highly incentivised to squeeze
customers financially as hard as possible; government IT spending is already
wasteful without propping up the IBM share price.

There are few paths forward to improve such systems. Beyond the lack of
skilled personnel, there's no real open source community or any community
producing libraries or integrations.

The problems with these sorts of rewrites are that non-technical managers
don't really understand the concept of tech debt until it's way too late and
you have a situation like this one. There's no sense that constant improvement
should be expected, like what you find in the tech industry, indeed they
usually take pride in avoiding projects that are "technology for technologies
sake". And that _sounds_ very reasonable, right up until you discover people
have basic expectations that you can't fulfil in any plausibly fast manner, or
everyone who understands your system has retired and died.

So to justify any replatforming or upgrade project, it has to be done via
promising lots of new features as part of the rewrite and that screws with the
sequencing. Now it's not a tech upgrade project but rather a total upgrade of
many different things simultaneously, including often rigid business
processes.

------
jcrawfordor
Attributing these problems to COBOL is short sighted. While COBOL is obsolete
for good reasons, it does provide high performance and makes it relatively
easy to construct new batch-processing tasks quickly, due to its built-in data
model (a concept which is largely absent from modern programming languages but
does pop up from time to time in forms like MS LINQ - historically it was far
more common, see also MUMPS). Because COBOL applications run predominantly in
batch-mode they are also fairly robust and easy to troubleshoot compared to
many modern systems, isolating a problem being a matter of running the batch
jobs individually and finding the bad intermediary state, and recovering from
a failure being a matter of just restarting the job. Intermediate state is
almost always persisted to storage so it's easy to work back to where the
problem occurred, unlike in e.g. a microservice architecture where doing so
can require a dedicated tracing system and still be quite difficult.

Consider that we could write a headline like "Amazon runs on a 50-year-old
operating system, and now it's falling apart." The sentence is not entirely
untrue but rather misses the point.

The problem is this: these systems are old and, more significantly, have
received extremely limited maintenance and enhancement over their lifetimes.
This means that there are few people familiar with each system (a "bus factor"
problem) and, more broadly, that the talent-base of COBOL programmers has been
allowed to recede to near-zero. While a common modernization (or
"recapitalization," a term preferred in government contracting) route in the
tech industry might be to take a legacy system written in e.g. Java and
progressively reimplement it in e.g. Python with a simplified architecture, a
common recap route in government is to take a legacy system written in COBOL
or s/370 assembly and transition it to an s/370 emulator running on HPUX on
NonStop. I've been involved in such processes a couple of times. It has the
advantage of maintaining the reliability properties of the legacy system at
relatively low-cost since very little software work is involved. It has the
tremendous disadvantage of allowing the organization to continue to not touch
the software, with much of the work often outsourced to a vendor who need not
be very familiar with the codebase either

You certainly wouldn't write anything in COBOL in 2020, but COBOL itself does
have a robust and even fairly maintainable design. The problem is that any
application written 20 years ago or more, regardless of language, is going to
be tremendously difficult to modify to meet changing requirements and
increased demand. Unemployment insurance applications reaching six million per
week, for example, would strain a system designed using the most modern
methodology which was specified for a few hundred thousand. The difference is
that, with a newer system, there are likely to be in-house engineers or
vendors who are very familiar with the system and will be able to expand its
capacity more rapidly.

Another underlying problem is that legacy systems designed using a mainframe
methodology are intended for vertical scaling - to process more records you
buy a bigger machine (or expand the LPAR or what have you). While horizontal
scaling has its own disadvantages there's a reason it's so popular today -
it's a lot faster to buy more capacity from a cloud provider than to
requisition a larger mainframe^w midcomputer^w emulation host. There are few
vendors and government acquisitions are slow at the best of times, even under
emergency sole-source rules.

In a way the whole thing is ironic, because a lot of COBOL application
interfaces are faster than any software we use today. From the perspective of
a human operator they can be absolutely outstanding once you get over a
learning curve. However, the lack of any enhancements made to these systems
mean that requirements changes are often "patched in peopleware" if you will,
and you end up with clerks that can enter data an order of magnitude more
efficiently than with a likely modern solution, but they then have to send the
records to a virtual printer to be ingested into a Java EE solution... this
kind of "enterprise patching" rapidly adds up to an extremely inefficient
system.

------
jerzyt
What is far more scary than the old stuff our government is running in COBOL,
is the new stuff running in Excel. Excel became a de facto Common Business
Oriented Language (COBOL), accessibly to anybody. Old COBOL programs were
still developed with some software engineering discipline. Current Excel
spreadsheets have none of that. And, this doesn't apply to the government only
- private enterprise runs bazillions(?) untested, unverified spreadsheets.
Wish I could share what I've seen.

~~~
gHeadphone
That’s true. I used to work in one of the biggest banks in the world. More
than half the divisions were dependent on Excel. Every time there was an
audit, a huge bunch of spreadsheets were automated. By the time the automation
was complete, another hundred spreadsheets would have popped up. It was like
trying to spray weeds. I remember one spreadsheet that contained the only
formula for calculating risk in an almost billion dollar portfolio. Genuinely
scary stuff.

------
smcphile
My first computing job, many years ago, was maintaining a large, COBOL program
for an insurance company. I doubt the problem the American government is now
facing has much to do with COBOL in itself.

It probably has much more to do with the fact that these COBOL programs were
designed to run on IBM mainframe computers and to interact with various
proprietary IBM subsystems (with names like CICS, VSAM, VTAM, DB2, etc.) These
subsystems are non-trivial to learn and there’s been little/no incentive for
anyone new to learn them for the last (at least) twenty years.

Finally, back in the 80’s and 90’s, some programs were written in IBM
assembler language instead of COBOL, for example when there was a need for
greater speed. Understanding and maintaining (or porting these programs to a
non-mainframe environment) is also non-trivial.

I left the IBM mainframe world completely 20 years ago, switched over to
working only on Unix/Linux systems, did that for 20 years, and am now happily
retired. However I’m sure there are still lots of older IBM mainframe people
around to handle these problems, they just cost money.

~~~
gregjor
A lot of the problem stems from the databases. COBOL, and these legacy
systems, pre-date relational databases. They work with proprietary and custom
file formats, which COBOL was designed to work well with. Regardless of
technical debt or scaling problems, the data in these systems has tremendous
value, and converting it to a modern relational structure is a bigger job than
fixing some bugs in COBOL code.

------
sullyj3
Not a great headline really, C is nearly 50, but most people wouldn't consider
it a disaster for everything to be written in C. The problem isn't that COBOL
is old, it's that it's largely obsolete.

~~~
TomMarius
Who would not consider a information system application written in C a
disaster? Can't think of anyone who who would want to work on that.

I agree with your point though.

~~~
jatone
no, because literally every application except the most trivial touch c.

every major RBMS is c or c++.

you don't write frontend's with c/c++ but if you're interacting with a
datastore most are c.

the problem with c and c++'s age are legacy behaviors related to them we've
since determined are not beneficial to actual programs and make reasoning
about them harder.

~~~
jjeaff
Of course everything touches c. But it's mostly all open source stuff that is
maintained by a large community of coders and is mostly underlying
dependencies.

When you are developing software, you just want that stuff to work. You don't
want to have to dig around in the internals to figure things out.

That's why scripting languages have become so popular.

------
GnarfGnarf
The problem with rewriting the COBOL code base in a newer, sexier language, is
that this new language will itself become obsolete in ten or twenty years.
Then you're back to square one re-writing the app in another newer language.

So businesses faced with this problem have a choice: accept that their code is
written in an old language, and train programmers to learn it. Or pay big
money to convert their code into... a language that will soon be obsolete, in
an endless treadmill.

Isn't it better to be obsolete and not pay conversion costs, rather than be
obsolete and pay perpetual conversion costs?

Yes, programmers are more productive in newer languages. However, you are
never going to get rid of the old stuff, so you will always need some
programmers versed in every language, at every stage of your conversion
hamster wheel.

------
cafard
I think that The Onion should run an article: "Coders Shocked to Find
Accounting Systems Written in a Language Designed for Accounting Systems."

------
m463
It seems like the age of something is the argument.

I starting thinking of this and thought:

\- Much modern AI software depends on 63-year old fortran numerical methods
code.

\- the Brooklyn Bridge is 137 years old.

\- the White House is 220 years old

\- Aristotle was using logical reasoning over 2300 years ago.

------
gregjor
Now I know how to reply when someone posts “I have a week with nothing to do.
What language should I learn next?” COBOL.

Seriously, anyone with some programming experience can learn COBOL or any
other language in a few weeks. Maybe with so many hip startups for concierge
cat sitting closing down more programmers with time on their hands and no
paycheck will crack the McCracken book (the one I learned COBOL from ages
ago).

~~~
non-entity
If i could get hired with a good rate after just learning the COBOL language
maybe. Unfortunately, I don't have a mainframe for me to learn the rest of the
environment in.

~~~
lboc
IBM give away free year-long z/OS accounts:

[https://news.ycombinator.com/item?id=20873531](https://news.ycombinator.com/item?id=20873531)

------
throwaway888abc
Informative comments. Thanks!

------
_bxg1
*60-Year-Old software systems

