Hacker News new | comments | ask | show | jobs | submit login

The IT 'cowboys' didn't ride into sunset.

They were fired by banks which are/were stupid enough to cut corners on people who know things about their infrastructure before migrating to more modern systems.


>One COBOL programmer, now in his 60s, said his bank laid him off in mid-2012 as it turned to younger, less expensive employees trained in new languages.

>In 2014, the programmer [...] was brought in as a contractor to the same bank to fix issues management had not anticipated.

Also FTA:

>Accenture’s Starrs said they go through a “black book” of programmer contacts, especially those laid off during or after the 2008 financial crisis.

>The job ultimately took five years and cost more than 1 billion Australian dollars ($749.9 million).

So, in short: a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job.

The 'problem' is entirely self-made.

> a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job

"a bank" is technically incorrect. The decision was a perfectly rational prisoner's dilemma defection from one executive (cut costs by firing the "expensive employees" and hiring "cheaper and newer"), who went on to be promoted on that basis, and it was the next generation of executives who had to deal with the mess and "hired IBM" so they could absolve responsibility of a situation with no visible upside if successful and tons of visible downside if not.

Except that's not what happened in the specific example given - it was a strategic decision to spend a billion dollars and multiple years to replace an old system that processed transactions in daily batch jobs with a modern system that can do transactions effectively instantly - which is exactly what the parent post was arguing should occur

In your statement you omitted the fact that they fired the very people they need to bring this "strategic decision" to life.

Spending money on upgrades is not the problem. Firing people who know about your systems is.

> cut costs by firing the "expensive employees" and hiring "cheaper and newer"

You should do everything else to cut costs before you start getting rid of your people. Those people understand your business and you're going to need those people when things turn around (and they usually do). Any time you let people go, you're throwing away domain knowledge. You've PAID for that. It is an asset. You'd no more throw away all the computers in the building, why would you throw away your employees!?

For the large companies, it was probably due to pensions. When you take into account ever increasing lifespans and the number of people nearing eligibility for pensions at that time, it was a no brainer for finance and management. A lot of people didn't get fired. Instead they were transferred to one of the big consulting firms like Accenture and they ended up working for the same pay but much less benefits as contractor to their former employer. Some of those affected were smart enough to sue their former employers but I would guess that most did not.

It is the banks who put in the perverse incentive systems. (Pay half of this year’s gross profits in End of Year bonuses)

You added a wrinkle that seems to partially explain it. Moreover, it does not and should absolve the relevant parties of their responsibility.

The ironic thing about your reply is that it is crafted in exactly the type of management speak that executives used to convince stakeholders in both occasions.

"absolve relevant parties of their responsibilities" - what relevant parties? What responsibilities? It sounds great and responsible, giving a good impression of dismissal of the point made, but the wording is so general as to be impossible to act upon - just as with "a bank". I'm sure standardised checklists were ticked, committees reviewed the decisions and came to the conclusion that it was best, the board was satisfied with the new direction taken by the CTO's organisation, and the shareholders happy with the cost reduction at the time as described by the board. To paraphrase Thatcher, what colour trousers do the relevant parties wear?

You see this defection pattern everywhere. For example, the never ending stream of useless new features in most Google and some Microsoft products which coincidentally get slower and buggier every year, because you can get promoted for the visible addition of a feature as a PM, but it is much harder to prove that getting the Gmail app load time down by 70% has stopped the feeding of your best power users to the competition (Calendar is too easy a target for criticism).

The classic defector on the tech side is the "ninja rockstar developer" whose prolific output of code and visually attractive but functionally superficial features is matched only by the amount of frustration generated after they are promoted somewhere else or changes jobs if it doesn't happen fast enough, and the rest of the team has to deal with the technical debt and new, unrealistic user expectations.

Security is another good candidate for shortcuts, since there is no visible upside to good security (maybe you were just not attacked) and tons of downside for taking responsibility since you make yourself a visible scapegoat for blame when a crisis does happen.

And note that none of these examples are in "move fast and break things" i.e. "do illegal things and hope they become legal ex-post" territory! I personally take great pains to understand these dynamics so that I can avoid them in my own company, which is a matter of survival.

All very true. Just focus on the basics, stay in your lane, and eventually your company will win. It's a tough balancing act.

The bank and "IT Cowboy's" technology teams have probably spent years on "Ayne Rand's Train." Projects headed for failure. Everyone is concerned with avoiding/pinning blame instead of building software that works. Deep knowledge of contracts instead of deep knowledge of code.

These are common pathologies. I agree though. This is incompetence at technology from strategy to the details.

It's important to understand how much technology gets built this way. Usually these projects are not outright, public failures.

The problem is that these decisions are rational - you get personally rewarded for getting yourself promoted or putting neat projects on your resume (which will increase your salary in your next job), but not for helping the business long-term. Avoiding blame is far more personally valuable, in terms of reward for time, than building good things.

If you take "I swear by my life and my love of it that I will never live for the sake of another man, nor ask another man to live for mine" seriously, this is the end result. Why should any IC or low-level manager at Taggart Transcontinental live for the sake of the Taggarts? But how can the railroad succeed without ICs and low-level managers?

If you want to fix this, don't ask people to live altruistically; instead, fix their incentives. Giving people cool projects such that they feel personally fulfilled when they succeed whether or not the business recognizes it is one option, but not particularly reliable or scalable.

Im not buying her interpretation of why the train scene happens either, but I do think the scene was kind of a brilliant metaphor for this sort of thing.

Whenever you have a systemically pathological situation like big, enterprise software projects often get into... it's probably an incentive problem.

Tldr agreed

I was on that project as a grad.

They spent a billion dollars, over 5 years, with a staff of 1,000 strong to replace old software.

I would be surprised if even 20 of the 1,000 were touching COBOL.

I'm guessing it was an Accenture project based on the outtakes.

That's what Accenture does: armies of cheap people for huge contracts. Flip a coin on the outcome. Not that IBM or any of the other top-level consultancies/services behemoths are any good, but Andersen Consulting/Accenture basically invented that approach.

It takes 980 MBAs to reach the level of schmoozing required to secure the contract.

    > The 'problem' is entirely self-made.
And richly deserved. The finance sector can afford it, for now.

"Technical debt is your job." -- is what I once heard some PHB say. Not from a finance person but the attitude, I think, is the same.

Which is a bad attitude in general, but an especially bad one for an industry known to use exceedingly old tech.

In all honesty: why is COBOL so prevalent in the finance sector? It can't seriously be SO GOOD as to be completely un-replacable?

Banks were among the first to computerize their systems, and switching systems cost a lot of money, regardless of industry. Hence they have a lot of old systems.

Because financial sector is run be people who consider engineers a liability[1]. Instead of evolving with time in 80s and 90s as the world moved to modern systems, they clung to their increasingly legacy systems. After all, why would an MBA toting executive from a big bank bother with something as menial as technology? At financial sector firms, Engineers are not meant to be seen or heard. Try proposing a new idea — or a move to a newer system — you’ll be shot down faster than you can say technology — by the same people who have led us from one financial crisis to another in their infinite wisdom.

Most big banks have a bunch of security people whose sole job is to make people’s life hell by spreading as much FUD as possible[2] — that’s what keeps their jobs secure, after all. And they are the only “engineers” who are heard. They make sure that anything new is not adopted till the rest of the world has already started abandoning it.

The primary focus in the technology departments of big banks is to maintain the status quo. New development is rare. New ideas are crushed before they can see the light of the day. As a result, good engineers leave as son as they can and the only ones left behind are the ones who are content living a life of mediocrity — maintaining the status quo.

[1] I know someone who was told something along those lines when they joined a big financial institution as an engineering manager. And not all financial companies work that way. Those who embraced technology reaped the benefits and continue to do so.

[2] I am not suggesting that security is not important. But security people at these institutions take it to a whole different level — to the point where things that should not take more thane a few days end up taking months — if they allow it, that is. Funny how all their objections vanish when someone from upper management asks for the same thing.

It applies to any old project.

Imagine that you need to replace a live system, downtimes are not allowed, probably provide a new UI while offering 100% of existing features.

There is no documentation available, or even if there is one, you won't find anyone in the building that will put their hand on the fire for how much of it actually maps to the running application.

So a project of reverse engineering existing application into a requirements document or updating the existing documentation usually takes several months.

Now just for a very basic project cost, consider the monthly salary of everyone involved, multipled by the amount of months.

And we are just speaking of phase I, then there is the actual re-write, followed by verification that 100% of the old features are still present on the new system.

Naturally it is cheaper to keep existing systems running.

This applies to any old platform, not only COBOL systems.

Except for the detail that COBOL programmers are outrageously hard to find, thusly expensive, and are only continuing to get harder to find and more expensive.

Except that it still cheaper to pay a pile of bucks to IBM and Unisys to keep those systems running than re-write them.

Which by itself shows how much money such re-write would eventually cost.

And in some cases, the source code is lost.

Replace COBOL with Visual Basic, Java, Javascript and you have your answer. It's what was used when those finance applications were written and became part of the technical debt of the institutions that wrote them. Your shiny new language/framework today may one day be fondly remember by future generations as 'why are we still using this crap?'

No one who's in a position to decide about COBOL's replacement has any strong incentives to do so. You don't want to be the one exec who made that risky move and crushed half their bank's tech foundation in the process.

why is COBOL so prevalent in the finance sector? It can't seriously be SO GOOD as to be completely un-replacable?

To use hip modern buzzwords, it’s a DSL. In addition, it runs on hardware specifically designed around its idioms.

It's not the language, it is the system that was built.

You can't just throw it away.

One of my jobs was for a subsidiary of a European plane maker. Some of their critical systems were written in Fortran 77 and was decades old in some cases. I have no clue about F77 or aircraft engineering, but my team was told repeatedly that these software ran flawlessly and nobody was ever to touch it (we made some fancy JAVA UI and called F77 executables in the background). I guess a time will come to sunset, but just not yet.

As above, may be these COBOL systems have been battle tested for a long time and they are good. Other reasons could be, "if it ain't broke then why fix it?" or that there aren't enough people to act as a bridge between the old and new systems so it is hard to do such projects (as other comments have pointed out)

> In all honesty: why is COBOL so prevalent in the finance sector? It can't seriously be SO GOOD as to be completely un-replacable?

It is not un-replacable in itself, I'd imagine it is just a huge undertaking and many higher ups are afraid to do it and risk something going wrong.

It's not unreplaceable, but it costs money to replace, and it's a risk to replace it; the old system is tried and tested, whereas the replacement might not work quite the same way.

So at any specific time, the safer and cheaper option is to stick with the old system, despite the growing risks and costs in the long run.

"If it's not broken, don't fix it" is a saying for a reason. This might be it.

It isn't just banks either. I'm a greybeard sysadmin type who has seen the insides of hundreds of companies, and for some reason in the late 2000s it seemed like everyone from law firms to insurance companies decided to fire their IT teams and then pay 3x the money for 1/16 the service from contractor/msp-types... and they wonder why they wallow in tech debt... I'd they even knew what it is, they did fire all the people telling them about it after all!


Operational expenses looks better on the balance sheet (ie. renting versus owning).

You don’t own employees.

MSP and Cloud are separate concepts. The typical MSP has a bunch of technicians who drive around to client sites and tend to the servers in their closets.

And, this shit has been going on for 20 years, my uncle (now in his 80's) made bank Cobol contracting and subcontracting EXACTLY the same way in the run up to Y2K: hiring developers "early retired".

The problem?

I wonder whether it was the same VP who got promotion for decision to cut corners a couple of years back, and another promotion for decision to fix it by contractors.

Pardon my ignorance but FTA means 'from the article', yes?

Yup, just another needless TLA.

you are correct

While most of your points are valid, you've misunderstood the last quote. CBA didn't fire people and then need to pay a billion dollars to fix something. They specifcally did migrate to a modern system - they completely replaced their entire legacy core banking system with a brand new one.

Self made ... by both the banks and the programmers.

As the article notes, these developers don't seem to have done a great job of documentation. Perhaps deliberately, given the comment about personal vindication. "You can't survive without me" is not a great look for either employee or employer.

Perhaps they are cowboy developers in more ways than one.

Right. Those developers who were given days and months to document their work deliberately did nothing instead.

Wait, no, I've literally NEVER heard of management asking developers to delay the next release to document their work.

Couldn't they just find the time to document while they're developing?

Oh, but they can't increase the estimate. The customer signed off on that estimate, so we're committed to deliver by that deadline.

Not really. Code needs to solidify before you document it. Documentation is best saved for after a major release.

Exactly. I'm always surprised when people think that it's easy to find time for documentation and that you only skipped that step because you are lazy and/or unprofessional.

Honestly, I know that's the conventional wisdom, but I've never once heard an engineer go to their manager and say, "I'd like to spend some time documenting this important process you asked me to implement" and being told "no, we don't have time to document! just keep it all in your head!"

I'm sure this literally never happens.

What I do see quite frequently is developers who don't like writing documentation finding ways to not do it or justify doing a poor job.

Only time I've encountered a desperate fixation by a CIO on documenting applications was when the relevant development team was about to be laid off.

Fortunately I had already resigned at that point.

You, obviously, have no idea how hard it is to write decent documentation for any system. Especially when it has been in existence for many years.

Good documentation is hard and anyone who thinks it is not does not have a clue as to what is needed for the purposes of maintaining any piece of software.

It is a very rare event to see documentation of a quality that explains all the interrelationships, reasons for the choice, etc, etc, etc.

I have in the past been required to provide all the documentation for projects I have worked on, functional and technical specifications, user guides and cheat sheets, detailed design documents and all of these take considerable time to write well. I can't say I am expert in this area because I am not. But I have done a lot better than most of the projects that I have had to interface to over the decades.

Documentation is a low priority for many management teams, they do not see the use in the kind of details that are needed. They do not give it sufficient support to do the job well.

I have seen and worked with people who have the skill to write the kind of documentation that is actually needed and they are a rare breed, well worth any money you pay them.

When companies complain about "cowboys", they often do not see that it is they who are the "cowboys".

You, obviously, have no idea how hard it is to write decent documentation for any system. Especially when it has been in existence for many years.

I've been programming for decades and my current project is frequently praised by other developers who work with it (it's a framework) for the quality of its documentation.

So, in fact, I do have such an idea. It's hard work. We do it anyway because we're professionals. The fact that people frequently don't do it doesn't change this.

One technique I use is to keep documentation in the same repository as the code, and force people to write docs via the code review process. That is, a proposed change doesn't pass review until it also updates the documentation files. People want to get their code merged, so they know they'll have to take docs seriously or else it'll get picked up in review.

Now maybe back in the mainframe days there was no culture of code review, but these are banks we're talking about. They excel at process. So it seems kind of unlikely they'd not want things documented, if management had understood what needed to be documented in the first place. But ultimately it was only the employees who knew that.

Your current project is praised for its documentation, that's actually good. However, if you are taken out by a bus today in some permanent fashion, how easily or quickly can someone come up to speed in supporting your project and not have to try an puzzle out why you did things the way you did? What assumptions are there that you have not documented? What relationship to other pieces of code or system functionality is there that you have not covered? What changes cannot be done because of the "brittleness" of interrelationships between different sections of the code? What are the expected business rules that are applicable to the software? What constraints do they put on others who want to use the software? What are the meanings of all the magic numbers that are littered throughout your code? Keep in mind all those other questions that they will have along the way.

It doesn't matter how "professional" you might be, you will not have covered all your bases. All you can do is spend lots of time in building up this documentation and if it is not being paid for in some way or time constraints get in the way, then none of it will get done properly.

It is good to see that you have some sort of process in place to manage the problem, but code reviews are limited in what they will capture.

I am documenting a piece of my own code at the moment for the purposes of submitting it to a specific project. It provides an additional functionality relating to something that the main system doesn't (at this point) handle. It is intended to be a simple workaround until the main system can supply this functionality at the VM and compiler level.

I have just looked at one method that simply converts a hexadecimal number to a potentially multi-byte sequence. The detailed reasoning for the code (the assumptions, choices of numeric values used, etc) amount to 10 times the number of lines that is actual code and I still don't think it is complete. I think that it will take at least two, three or maybe even four more revisions in the documentation to get it to the place where someone else could make changes without causing problems elsewhere. The code itself is really simple. But, there is an additional purpose to this documentation, in that, it will provide a baseline against which the final system changes can be compared.

I do this because, as a somewhat retired "professional" programmer, I want to see some serious documentation for this system and can afford to put in the time to do so.

So back to the main discussions, unless full and complete documentation is a requirement put in place and strongly enforced, what documentation there is will not be sufficient for the future. It is a rare professional who will instigate this and make sure it happens. From my experience, such professionals that do this, have not come from a programming background but have come from other industries (usually engineering based) where such documentation is considered normal or is mandatory.

I have seen few management teams that have given more than lip service to creation of the required documentation and the time frames they put on projects are such that there is little or no time for the production of quality documentation that is needed. Nor are they interested in paying for such.

Banks have a focus of making money. They are not interested in spending it. The only process orientation they have is increasing the bottom line in whatever way they can. If it means they cut back on programmer and computing costs, they will do this. Ultimately, it is the bank management that holds responsibility to understand that the documentation is required. If they did not understand this, they should not have been in the position of management. It is irrelevant if only the employees (those grunts at the coal-face) knew this.

If you want to blame someone, you can always say that he didn't document his stuff enough.

The problem is, that there is no standard, what's "enough", so it is easy to move the goal posts. Combine that with fast paced environment and putting out fires constantly, and you have a recipe for disaster.

>If you want to blame someone, you can always say that he didn't document his stuff enough.

This, he could have documented it perfectly well in a time when everyone knew COBOL.

But that documentation would all be useless to me today with my knowledge of Java and Kotlin, but understandable to someone else from is generation that works with COBOL.

Tbf, it only really makes sense for the individual programmer to properly document code if there’s a decent chance they’ll forget (eg some obscure part of the codebase), or if you expect a reasonable chance that someone will have to read the code without you. Otherwise, its useful, but not necessary: you just go to the guru (perhaps yourself); his living body is the documentation.

If you’re thinking that only you’ll ever read it, or you’ll be working on this codebase for the next 30 years, it doesn’t really matter. Peoper Documentation is a cultural activity required for teams, especially those with shifting workers, as is common now. And should really be enforced by the managers, not the programmers thenselves.

Its significantly less necessary when you can expect to spend most of your career on a single codebase, a workstyle these cobol programmers presumably came from.

I try to design and name interfaces clearly and comment code for myself. If it's been a few weeks since I've looked at something it helps me snap back. Also, I would never approach a codebase as if I will be working on it for 30 years. Job security comes from marketability, not obscurity.

Was that true when these systems were first made? Many of them are decades old, and the culture of large institutions is usually resistant to change, and that comes from management on down. If the system didn't start with those values in mind, what would have prompted them to change?

It's not that I don't expect the codebase to age or design it with long-term thinking, I just don't expect to be working on it for that long. Someone will replace me, and I'd like to leave them with well-designed, readable code that they will be able to understand.

I'm talking about that as well. The modern era of jumping ship every 2-3 years on average in software is not how its always been. Jobs used to be very long term

It's not always about jumping ship either. In many companies, you will move between areas/needs or get promoted to other work after a couple years. Also, companies will outsource maintenance to lower-cost teams. It seems there is better project lifecycle management, less waterfall development, and fewer cushy open-ended government contracts. But, you're right. The expectation that you'll be able to do the same thing indefinitely for you career is gone.

This is a terrible way to maintain a system. You never know when the guru is going to die or quit; you have to have a way of getting other people to maintain the system. This is why humans invented written language: to transmit information to each other and to subsequent generations.

However, writing useful documentation isn't trivial, and is best done with a technical writer, and takes time away from development. If management isn't willing to invest the time and resources necessary to produce useful documentation, then they're going to have problems in the future as the gurus leave.

What happens when a bus hits the guru?

Guru meditation error.

Your DR planning better include a necromancer.

yeah. this is how it had been done before the invention of writing. time has proven it not to be a very efficient method.

The only way to have really quality documentation is to have a lot of people reading it and providing feedback, over time.

It also doesn't hurt when a quality technical writer actually writes or edits the documentation in consultation with the people who designed or implemented the system.

How many new devs are willing to even touch a COBOL program. documentation or not.

For enough money, I'd say many. COBOL isn't sexy, but it's not difficult either. And it's demonstrated a lot more staying power than most other languages.

Darn right! I'd cheerfully learn Cobol for a job with the right parameters. I once took an RPG course in order to apply for one (though sadly did not get it).

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact