They were fired by banks which are/were stupid enough to cut corners on people who know things about their infrastructure before migrating to more modern systems.
>One COBOL programmer, now in his 60s, said his bank laid him off in mid-2012 as it turned to younger, less expensive employees trained in new languages.
>In 2014, the programmer [...] was brought in as a contractor to the same bank to fix issues management had not anticipated.
>Accenture’s Starrs said they go through a “black book” of programmer contacts, especially those laid off during or after the 2008 financial crisis.
>The job ultimately took five years and cost more than 1 billion Australian dollars ($749.9 million).
So, in short: a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job.
The 'problem' is entirely self-made.
"a bank" is technically incorrect. The decision was a perfectly rational prisoner's dilemma defection from one executive (cut costs by firing the "expensive employees" and hiring "cheaper and newer"), who went on to be promoted on that basis, and it was the next generation of executives who had to deal with the mess and "hired IBM" so they could absolve responsibility of a situation with no visible upside if successful and tons of visible downside if not.
Spending money on upgrades is not the problem. Firing people who know about your systems is.
You should do everything else to cut costs before you start getting rid of your people. Those people understand your business and you're going to need those people when things turn around (and they usually do). Any time you let people go, you're throwing away domain knowledge. You've PAID for that. It is an asset. You'd no more throw away all the computers in the building, why would you throw away your employees!?
"absolve relevant parties of their responsibilities" - what relevant parties? What responsibilities? It sounds great and responsible, giving a good impression of dismissal of the point made, but the wording is so general as to be impossible to act upon - just as with "a bank". I'm sure standardised checklists were ticked, committees reviewed the decisions and came to the conclusion that it was best, the board was satisfied with the new direction taken by the CTO's organisation, and the shareholders happy with the cost reduction at the time as described by the board. To paraphrase Thatcher, what colour trousers do the relevant parties wear?
You see this defection pattern everywhere. For example, the never ending stream of useless new features in most Google and some Microsoft products which coincidentally get slower and buggier every year, because you can get promoted for the visible addition of a feature as a PM, but it is much harder to prove that getting the Gmail app load time down by 70% has stopped the feeding of your best power users to the competition (Calendar is too easy a target for criticism).
The classic defector on the tech side is the "ninja rockstar developer" whose prolific output of code and visually attractive but functionally superficial features is matched only by the amount of frustration generated after they are promoted somewhere else or changes jobs if it doesn't happen fast enough, and the rest of the team has to deal with the technical debt and new, unrealistic user expectations.
Security is another good candidate for shortcuts, since there is no visible upside to good security (maybe you were just not attacked) and tons of downside for taking responsibility since you make yourself a visible scapegoat for blame when a crisis does happen.
And note that none of these examples are in "move fast and break things" i.e. "do illegal things and hope they become legal ex-post" territory! I personally take great pains to understand these dynamics so that I can avoid them in my own company, which is a matter of survival.
These are common pathologies. I agree though. This is incompetence at technology from strategy to the details.
It's important to understand how much technology gets built this way. Usually these projects are not outright, public failures.
If you take "I swear by my life and my love of it that I will never live for the sake of another man, nor ask another man to live for mine" seriously, this is the end result. Why should any IC or low-level manager at Taggart Transcontinental live for the sake of the Taggarts? But how can the railroad succeed without ICs and low-level managers?
If you want to fix this, don't ask people to live altruistically; instead, fix their incentives. Giving people cool projects such that they feel personally fulfilled when they succeed whether or not the business recognizes it is one option, but not particularly reliable or scalable.
Whenever you have a systemically pathological situation like big, enterprise software projects often get into... it's probably an incentive problem.
They spent a billion dollars, over 5 years, with a staff of 1,000 strong to replace old software.
I would be surprised if even 20 of the 1,000 were touching COBOL.
That's what Accenture does: armies of cheap people for huge contracts. Flip a coin on the outcome. Not that IBM or any of the other top-level consultancies/services behemoths are any good, but Andersen Consulting/Accenture basically invented that approach.
> The 'problem' is entirely self-made.
"Technical debt is your job." -- is what I once heard some PHB say. Not from a finance person but the attitude, I think, is the same.
In all honesty: why is COBOL so prevalent in the finance sector? It can't seriously be SO GOOD as to be completely un-replacable?
Most big banks have a bunch of security people whose sole job is to make people’s life hell by spreading as much FUD as possible — that’s what keeps their jobs secure, after all. And they are the only “engineers” who are heard. They make sure that anything new is not adopted till the rest of the world has already started abandoning it.
The primary focus in the technology departments of big banks is to maintain the status quo. New development is rare. New ideas are crushed before they can see the light of the day. As a result, good engineers leave as son as they can and the only ones left behind are the ones who are content living a life of mediocrity — maintaining the status quo.
 I know someone who was told something along those lines when they joined a big financial institution as an engineering manager. And not all financial companies work that way. Those who embraced technology reaped the benefits and continue to do so.
 I am not suggesting that security is not important. But security people at these institutions take it to a whole different level — to the point where things that should not take more thane a few days end up taking months — if they allow it, that is. Funny how all their objections vanish when someone from upper management asks for the same thing.
Imagine that you need to replace a live system, downtimes are not allowed, probably provide a new UI while offering 100% of existing features.
There is no documentation available, or even if there is one, you won't find anyone in the building that will put their hand on the fire for how much of it actually maps to the running application.
So a project of reverse engineering existing application into a requirements document or updating the existing documentation usually takes several months.
Now just for a very basic project cost, consider the monthly salary of everyone involved, multipled by the amount of months.
And we are just speaking of phase I, then there is the actual re-write, followed by verification that 100% of the old features are still present on the new system.
Naturally it is cheaper to keep existing systems running.
This applies to any old platform, not only COBOL systems.
Which by itself shows how much money such re-write would eventually cost.
To use hip modern buzzwords, it’s a DSL. In addition, it runs on hardware specifically designed around its idioms.
You can't just throw it away.
As above, may be these COBOL systems have been battle tested for a long time and they are good. Other reasons could be, "if it ain't broke then why fix it?" or that there aren't enough people to act as a bridge between the old and new systems so it is hard to do such projects (as other comments have pointed out)
It is not un-replacable in itself, I'd imagine it is just a huge undertaking and many higher ups are afraid to do it and risk something going wrong.
So at any specific time, the safer and cheaper option is to stick with the old system, despite the growing risks and costs in the long run.
Operational expenses looks better on the balance sheet (ie. renting versus owning).
MSP and Cloud are separate concepts. The typical MSP has a bunch of technicians who drive around to client sites and tend to the servers in their closets.
I wonder whether it was the same VP who got promotion for decision to cut corners a couple of years back, and another promotion for decision to fix it by contractors.
As the article notes, these developers don't seem to have done a great job of documentation. Perhaps deliberately, given the comment about personal vindication. "You can't survive without me" is not a great look for either employee or employer.
Perhaps they are cowboy developers in more ways than one.
Wait, no, I've literally NEVER heard of management asking developers to delay the next release to document their work.
Oh, but they can't increase the estimate. The customer signed off on that estimate, so we're committed to deliver by that deadline.
I'm sure this literally never happens.
What I do see quite frequently is developers who don't like writing documentation finding ways to not do it or justify doing a poor job.
Fortunately I had already resigned at that point.
Good documentation is hard and anyone who thinks it is not does not have a clue as to what is needed for the purposes of maintaining any piece of software.
It is a very rare event to see documentation of a quality that explains all the interrelationships, reasons for the choice, etc, etc, etc.
I have in the past been required to provide all the documentation for projects I have worked on, functional and technical specifications, user guides and cheat sheets, detailed design documents and all of these take considerable time to write well. I can't say I am expert in this area because I am not. But I have done a lot better than most of the projects that I have had to interface to over the decades.
Documentation is a low priority for many management teams, they do not see the use in the kind of details that are needed. They do not give it sufficient support to do the job well.
I have seen and worked with people who have the skill to write the kind of documentation that is actually needed and they are a rare breed, well worth any money you pay them.
When companies complain about "cowboys", they often do not see that it is they who are the "cowboys".
I've been programming for decades and my current project is frequently praised by other developers who work with it (it's a framework) for the quality of its documentation.
So, in fact, I do have such an idea. It's hard work. We do it anyway because we're professionals. The fact that people frequently don't do it doesn't change this.
One technique I use is to keep documentation in the same repository as the code, and force people to write docs via the code review process. That is, a proposed change doesn't pass review until it also updates the documentation files. People want to get their code merged, so they know they'll have to take docs seriously or else it'll get picked up in review.
Now maybe back in the mainframe days there was no culture of code review, but these are banks we're talking about. They excel at process. So it seems kind of unlikely they'd not want things documented, if management had understood what needed to be documented in the first place. But ultimately it was only the employees who knew that.
It doesn't matter how "professional" you might be, you will not have covered all your bases. All you can do is spend lots of time in building up this documentation and if it is not being paid for in some way or time constraints get in the way, then none of it will get done properly.
It is good to see that you have some sort of process in place to manage the problem, but code reviews are limited in what they will capture.
I am documenting a piece of my own code at the moment for the purposes of submitting it to a specific project. It provides an additional functionality relating to something that the main system doesn't (at this point) handle. It is intended to be a simple workaround until the main system can supply this functionality at the VM and compiler level.
I have just looked at one method that simply converts a hexadecimal number to a potentially multi-byte sequence. The detailed reasoning for the code (the assumptions, choices of numeric values used, etc) amount to 10 times the number of lines that is actual code and I still don't think it is complete. I think that it will take at least two, three or maybe even four more revisions in the documentation to get it to the place where someone else could make changes without causing problems elsewhere. The code itself is really simple. But, there is an additional purpose to this documentation, in that, it will provide a baseline against which the final system changes can be compared.
I do this because, as a somewhat retired "professional" programmer, I want to see some serious documentation for this system and can afford to put in the time to do so.
So back to the main discussions, unless full and complete documentation is a requirement put in place and strongly enforced, what documentation there is will not be sufficient for the future. It is a rare professional who will instigate this and make sure it happens. From my experience, such professionals that do this, have not come from a programming background but have come from other industries (usually engineering based) where such documentation is considered normal or is mandatory.
I have seen few management teams that have given more than lip service to creation of the required documentation and the time frames they put on projects are such that there is little or no time for the production of quality documentation that is needed. Nor are they interested in paying for such.
Banks have a focus of making money. They are not interested in spending it. The only process orientation they have is increasing the bottom line in whatever way they can. If it means they cut back on programmer and computing costs, they will do this. Ultimately, it is the bank management that holds responsibility to understand that the documentation is required. If they did not understand this, they should not have been in the position of management. It is irrelevant if only the employees (those grunts at the coal-face) knew this.
The problem is, that there is no standard, what's "enough", so it is easy to move the goal posts. Combine that with fast paced environment and putting out fires constantly, and you have a recipe for disaster.
This, he could have documented it perfectly well in a time when everyone knew COBOL.
But that documentation would all be useless to me today with my knowledge of Java and Kotlin, but understandable to someone else from is generation that works with COBOL.
If you’re thinking that only you’ll ever read it, or you’ll be working on this codebase for the next 30 years, it doesn’t really matter. Peoper Documentation is a cultural activity required for teams, especially those with shifting workers, as is common now. And should really be enforced by the managers, not the programmers thenselves.
Its significantly less necessary when you can expect to spend most of your career on a single codebase, a workstyle these cobol programmers presumably came from.
However, writing useful documentation isn't trivial, and is best done with a technical writer, and takes time away from development. If management isn't willing to invest the time and resources necessary to produce useful documentation, then they're going to have problems in the future as the gurus leave.
It also doesn't hurt when a quality technical writer actually writes or edits the documentation in consultation with the people who designed or implemented the system.