
Banks scramble to fix old systems as IT 'cowboys' ride into sunset - petethomas
http://reuters.com/article/technologyNews/idUSKBN17C0D8
======
sanguy
Been working on accounting systems in RPG and COBOL since ~1992. I also know
C/X86ASM/Pascal/Delphi/VB/Fortran. Never bothered with C++ that much; played
with Java a bit but Oracle irritates my bowels so moved away from that.

As mentioned in the article it's good work; but it is also not easy work. You
tend to go through cycles of being pushed out to brought back under extreme
emergency at any costs to get stuff working. Only for the cycle to repeat.
Companies never think of the old guys as the ones to implement the new system
- that's a job for the "enterprise experts" \- I can't even keep track of how
many "rewrites" I've seen in my life fail because of this.

We are the dinosaur club; but it's a club that pays extremely well (high 6
figures a year without working too hard if you are talented and have a good
client base and reputation), but like fossil fuel one day it will all be gone
;)

~~~
Joeri
> Companies never think of the old guys as the ones to implement the new
> system - that's a job for the "enterprise experts"

Exactly this is why rewrites fail. The challenge of a rewrite is not in
mapping the core architecture and core use case, it's mapping all the edge
cases and covering all the end user needs. You need people intimately familiar
with the old system to make sure the new system does all the weird stuff which
nobody understood but had good reasons that was in the corners of the old
system's code. IMHO the best way to approach a rewrite is to make a blended
team of experts of the old system and experts of the new technology, and you
put a manager in charge with excellent people skills who can get them to work
together.

~~~
z3t4
all features and edge cases should have a test. if possible also have tests
for all the bugs ever found in the old system. it helps if the tests are not
too coupled with the code and are well documented. a test could be making an
entry where deb and cred dont match. or trying to enter an extra decimal. and
if the system adds two decimals corectly.

~~~
mikehollinger
>all features and edge casss should have a test.

Yeah, but the interesting problem is what happens next. Let's say a test case
reveals that there's a flaw in the next-gen system. You fix it. It later turns
out that the same flaw exists in the legacy system. What do you do?

Do you revert the fix, or leave it in place?

~~~
cle
Revert the fix and add it to your backlog. You've got enough to worry about
when porting/rewriting, don't add additional dimensions of complexity and risk
by trying to change business logic at the same time. Minimize risk by
minimizing change...then when the new system is up on its feet, go back and
fix all the mistakes you found.

~~~
kwhitefoot
I upvoted that but I suspect that there will never be an opportunity to "go
back and fix all the mistakes you found".

At least there never is for me.

------
beat
I worked for years on the modernization of a critical banking infrastructure
system. The problems are manifold. The biggest problem is overall
architectural impedance mismatch - switching from a batch system (most of
those old COBOL mainframe apps are batch systems) to a service-oriented
architecture. This makes incremental replacement extremely difficult. But a
cutover? On a system that moves more money in a day than the GDP of most
countries? A failure could put the entire banking world in a tailspin, cause
an economic crash.

Next, there's a lot of business logic basically embedded in the COBOL, or even
at lower layers. For example, lots of banking files are in EBCDIC, a different
character set from ASCII. Except there are _lots_ of different EBCDIC
variants, and there's no good way to tell which one you're viewing just by
looking at the file. So you have to reverse-engineer COBOL to figure out the
"correct" meaning of a given file.

The problems go on and on. When I see people in the startup world rolling
their eyes at the "incompetence" of the enterprise world, I take it to mean
they've never actually worked on a truly hard problem in their lives.

~~~
wolfgke
> When I see people in the startup world rolling their eyes at the
> "incompetence" of the enterprise world, I take it to mean they've never
> actually worked on a truly hard problem in their lives.

I worked in a related field in the past. I don't claim that the problems are
easy - where I tend to roll eyes rather is:

\- Bureaucracy: A lot is required by law, I know. But there is a difference
between "just following the required bureaucracy in a minimal necessary way if
it stands in the path" (the startup way) vs. "taking it seriously in a way
that makes the work harder than strictly necessary".

\- Hierarchies: Just three words: I hate them.

\- Unwillingness (?) to tackle these hard problems: The problems are hard, as
you already outlined. But this implies to me that everybody in the company
will move heaven and earth so that the people who work on this hard problems
are able to (e.g. give all necessary information (requirement specification
etc.) they know of etc.). If there is just one thing (or even office politics)
which will prevent that, I don't just roll my eyes, but get furious. Because
of the hardness this is not to consider an obstacle, but targeted sabotage -
and should be considered as such.

~~~
ExactoKnight
Part of this comes from an inability in the early years to truly respect
engineering as both a craft and as a form.

I know your startup mindset well and I carry with it with me too. I came into
a legacy fintech company the same way and pushed for faster decision making
processes.

I didn't realize the cause for conservatism until I was given a story about
how the company needed to manually call and refund thousands of customers...
all because one developer fucked up and double charged people.

When you deal with MONEY and you experience getting burned like that... you
realize how mantra's like "move fast and break things" only work as convenient
motto's for startups who have nothing to lose.

------
StillBored
$100 an hour, so $208k a year (assuming no time off), for a guy with 30+ years
working on a critical system in an industry where floor traders can make 7-8
figures, and huge bonuses are paid to "analysts". I have plumber buddies that
make more than that per hour (and with a crew can make more a year).

That's why a couple years ago when I had access to a zseries, and was hearing
about how "desperate" the banks were I took a look. What I discovered are
salaries that are some of the lowest in the IT industry and a cynical attitude
towards hiring. AKA, banks would rather pay $40k a year for a "system
operator" which is generally the ground floor for learning anything about a
mainframe than hire someone with a comp-sci degree and teaching them the
system while promoting them.

So, no thanks (you can fill in a less polite version), the banks can go to
hell.

~~~
movedx
Not sure if the $100/hour figure quoted is a hard and fast cash amount banks
strictly adhere to, but ultimately this is business, so: set your price and
walk away if it's not met.

I set my price: AUD$850/day, with a TINY bit of wriggle room, and that's
final. I know my worth (from a skills perspective and market/economical one.)

~~~
nstj
You could probably charge higher - this is at the mid range of many client
side programmer daily rates in AU and systems skills like this are so much
more critical.

~~~
movedx
Maybe. I'm a DevOp, not a coder. I could charge more.

------
ams6110
The problem really isn't COBOL. COBOL is not a difficult language. It reads
almost like English sentences. Any competent programmer would not be
challenged by COBOL. The problem is all the glue. JCL, SNA, CICS, ISM, ISPF,
etc.

It's like the difference between Java (the language) and J2EE, except in the
context of 1970s computer technology. It's not intuitive and not something you
can really work through without a lot of training and experience.

~~~
clumsysmurf
At my last employer, there were a few "mainframers" (we called them). I was
curious so I would ask my mainframer coworker how things worked on his end.

I always struggled to understand, because it seemed everything was different
... the terminology, the culture, the ideas. I couldn't use analogy to tie
what he was describing back to what I knew.

~~~
nsxwolf
You got that right. I had to deal with an interesting banking file format and
I asked a question on Stack Overflow and got this gem of an answer.

I politely accepted it as the right answer but boy oh boy does it feel like
it's from an alien civilization.

[http://stackoverflow.com/questions/28640159/what-is-the-
diff...](http://stackoverflow.com/questions/28640159/what-is-the-difference-
between-a-variable-blocked-format-and-a-fixed-blocked-for)

~~~
defined
Although I was never a mainframe programmer per se, I did quite a bit of
interfacing between mini/microcomputers and IBM mainframes, so I got to see
under the hood a little. (If I write something stupid, it's either memory
issues or ignorance).

I recall seeing how files were allocated on disk (remember that mainframes
have many different OSes, like OS\390, and even OSes on top of OSes like
VM/CMS, and I don't remember what this was running on).

In this particular case, a file was preallocated in JCL to use N extents
starting on a specific cylinder. Fixed size. None of this fancy ext3 or NTFS
;)

JCL (Job Control Language) was a language to control batch jobs, and many have
called it the worst language ever designed, although not as bad as brainf __k.

On the other hand, I had a chance to interface C++ with CICS (a transaction
processing subsystem) using WebSphere MQ, and I must say, I was really
impressed with its sophistication. It was a kind of SOA long before the term
was invented.

A lot of what I saw in the mainframe world predated things - by decades - that
some may think are new(er) concepts, such as clusters (sysplex), front-end
processors, hypervisors, HA, and so on.

Those of us who had to fiddle with implied file formats with fixed-length
fields and records won't find this stuff quite as alien, but equally as
painful to deal with. I recall using some sort of ETL program to get around
this. On the plus side, these primitive formats certainly were efficient in
terms of processing speed, and a great match for COBOL.

Speaking of COBOL, as part of this project, I had to write a parser in C++ to
parse COBOL copybooks (kind of a COBOL data structure definition) and generate
C code to read the data.

It is a very different world, but I don't think it's all bad. After all, the
technology has been working very well for a long time. Kudos to the COBOL
Cowboys. I hope they charge a lot more than $100/hr!

~~~
skissane
> In this particular case, a file was preallocated in JCL to use N extents
> starting on a specific cylinder. Fixed size.

Sounds like a z/VSE system (formerly known as VSE/ESA, VSE/SP, DOS/VSE,
DOS/VS, DOS/360). In DOS JCL (which is a different syntax to z/OS / OS/390 /
MVS / OS/VS2 / OS/360 JCL), you manually allocate files to disk locations
using the EXTENT statement. By contrast, in z/OS the operating system decides
where on disk to locate your file (or dataset, to use mainframe terminology).
(You don't _have_ to manually allocate files any more in z/VSE – you can use
VSAM, or store your files in libraries, and in both cases the OS decides on
disk locations for you – but, originally, neither VSAM nor libraries existed,
so you had to manually assign locations to all the files on disk.) It is very
primitive, but remember it was designed in the 1960s to run on machines with
only 16KB of memory–plus, humans could design a disk layout to maximise
performance, by placing frequently used files on faster areas of the disk.
Nowadays, the OS can do a better job of locating files on disks than humans
can do, but this capability is kept for backward compatibility.

~~~
defined
Thanks for this! I had a number of interactions over the years with the S/3x0
world, and I wasn't always sure what was under the hood. I was aware that
there was a bewildering slew of xxAM access methods, but had no chance to look
into them.

------
janwillemb
> One COBOL programmer, now in his 60s, said his bank laid him off in mid-2012
> as it turned to younger, less expensive employees trained in new languages.
> [...] "The call back to the bank was something of a personal vindication for
> me," he said.

I find this being an oddly satisfying part of the story

------
contingencies
First employee/architect of Kraken here. (Now does >USD$10M+/day of volume by
my estimate)

Accounting systems of all kinds should be generic and open source. Why do we
need banks at all? It's not like this infrastructure really has any special
characteristics. Most of it even has downtime and batch jobs. Fact: The bulk
of banking is just recording some numbers at a stupendously simplistic
resolution, with maybe 64 characters of description and a date.

A few weeks ago I spent a day interviewing 20 different Hong Kong banks about
API availability for cross-border RMB transactions. None at all offered it.

 _We 're reaching a point where the financial systems of a mid-level company
exceed those of the banks they are forced to utilize._

Modern requirements include things like: 24x7x365 availability, multilingual,
arbitrary asset type support (energy, carbon credits, cryptocurrencies, space,
time, etc.), multi-asset type accounts, new settlement networks, real time
reporting and AML/KYC, all features API-available, new and established
customer interaction through non-snailmail/physical means, customers routinely
in different countries, multi-user accounts with disparate access levels (eg.
accountant/auditor/spouse/kid), multiple legal jurisdictions with clashing
regulatory frameworks, settled-means-settled, regulator-forced free market
integration for non-core (ie. account-related) financial services such as
loans/forex, redundant service provider availability for every function,
meaningful SLAs/reputation for service providers, routing and/or provider
selection based upon nontraditional metrics such as ethical investment
rationales, etc. The same set of requirements goes up and down the supply-
chain: people want to reason with their suppliers and customers about stock,
settlement status, payment and contracts, they sometimes need backups in case
of failure down the chain, and they care about reputation.

Frankly the whole area is such a mess I am expecting an open source core
accounting project to take over the sector. Probably it will begin in
smaller/developing world banks and move toward the big guys like a meteor.

For some evolving thoughts on the area (from 2012, but literally picked up
again in the last 2 days) see [http://www.ifex-project.org/our-
proposals/ifex](http://www.ifex-project.org/our-proposals/ifex)

~~~
gaius
_A few weeks ago I spent a day interviewing 20 different Hong Kong banks about
API availability for cross-border RMB transactions. None at all offered it._

They don't do it because doing it while complying with all relevant
regulations is more complicated than you realise. That's what banks are
ultimately in the business of, finding ways to do business across
jurisdictions. And it's why most fintech companies fail: their beautiful code
runs headlong into the messy, illogical, unpredictable real world of
regulatory compliance and guess what, the regulator always wins.

~~~
osullivj
Strongly agree based on my work in the financial sector since 2009.
Everything's driven by regulatory compliance now. To the point where RegTech
is a thing.

------
nodesocket
Cobol expert = $1,000 an hour rate. Great niche, but eventually will get
replaced.

Awesome marketing and PR for his company Cobol Cowboys
([http://cobolcowboys.com](http://cobolcowboys.com).

> His wife Eileen came up with the name in a reference to "Space Cowboys," a
> 2000 movie about a group of retired Air Force pilots called in for a
> trouble-shooting mission in space. The company's slogan? "Not our first
> rodeo."

~~~
hackerboos
We have 50,000 lines of RPG code we need to replace over the next decade.
Since the last RPG dev at our company is over 65 years old and can retire at
any moment.

It's going to be funny when we replace it with Java and our business guys ask
why it runs much slower.

~~~
tigershark
I never wrote RPG code, but 50k LOC seems a pretty small project to me...

~~~
dragonwriter
A 50k LOC program serving a critical business function without a test suite or
requirements documentation can be quite a chore to replace, especially if you
let it go until you have neither staff experienced in the language or with
experience with the background of the specific application.

~~~
WalterBright
I've converted such programs from one language to another. It is not necessary
to understand how the program works, only how the language works.

What you do is function by function, convert the language by duplicating what
it does in the new language. Resist any and all urges to fix it, refactor it,
improve it, etc. Just translate.

~~~
chii
The problem is if there's some obscure language behaviour that's not
documented well, and the code relied on that behaviour.

unless you're an expert and knows the language inside out, it's hard to even
know such a problem occurred.

~~~
WalterBright
True, but languages tend to be far, far better understood and documented than
some random application logic.

------
kingnothing
"Experienced COBOL programmers can earn more than $100 an hour when they get
called in to patch up glitches, rewrite coding manuals or make new systems
work with old."

Any contract-based engineer would charge that, or more, regardless of the
language. Fortune 500 companies are happy to pay $300 / hour to a consulting
company for engineering time.

~~~
inopinatus
I just assumed that $100/hr sounds like a lot to the journalist in question
and therefore "more than $100/hr" was simply their top categorisation of
compensation.

Bear in mind that this is something like four times the average in private
industry, and fifteen times the minimum wage.

My general rule for stories by journalists - even experienced ones - covering
any technical industry is that their use of facts is often extremely loose,
and if something seems awry, distrust the story first and question your
understanding of the domain second.

~~~
therealdrag0
My first job out of college I worked for a large-corp building a CRUD java
web-app for one of their clients who implemented large-corp's backend. They
billed the client 250/hr for my time (full time for 18 months). So yeah
anything close to 100/hr is leaving piles on the table.

------
dsr_
The problem is technical debt. Banks didn't realize they were software houses,
they thought that IT was a tool.

~~~
flukus
It is amazing how many companies don't think they're software companies, even
though they are 100% reliant on their in house software. "We aren't a software
company" is even a direct excuse I've heard a few times to justify ignoring
technical debt.

~~~
grogenaut
What's worse is companies who 100% outsource to customized versions of a
vendor's code. At least banks own the source to what they run even if they
can't run it without a license. Clothing company I know sitting next to amazon
pays the equivalent of about 5 fte senior amazon engineers a year in licensing
the software they run every year to a vendor who charges them another 5
engineers worth to implement all of the customizations they want and "re-
negotiates" the contract every 3 years. I say negotiate because there's not
really an alternative to switch, it'd be a 3 year conversion project and they
refuse to hire devs.

They gave a 8k raise to most tech folks about 5 years ago because they were
bleeding people across the street. Which brought their senior tech folk to
within 20k of an amazon SDE intern salary. People still left.

I'm biased as an engineer but I'm not really sure why they keep this
relationship going.

~~~
flukus
This is accountant math with capex vs opex. I don't know how but they manage
to convince companies that their saving money by spending 10 times as much.

~~~
eru
Might still be a good decision: the corporate politics get so much harder,
once you have in-house teams. And: that company would probably fail at writing
in-house software---most of them do.

~~~
grogenaut
Good point. I didn't think about the leverage of having an internal team
holding you hostage vs and external.

And yes there is also the, how do you hire up a good dev team when you
obviously don't know how to do software yourself. Eg hiring outside your
competency. I know this is especially hard, and I've watched large companies
who are good at hiring in general fail at it specifically in a wheel house. I
think this is why aqui-hires can help.

~~~
eru
Semi-related: that's why 'blockchain technology' can be so useful to banks.
It's not that they need anything blockchain at all, but in practice CEOs are
not actually all that powerful against the vested interests of middle
management, but having the excuse of hyped up 'blockchain technology' can help
with pushing through common-sensical reforms that you would want to do anyway.

What's a wheel house?

~~~
grogenaut
It's the core of a baseball batter's hitting area. Eg their core strength.

Challenge for you since you invoked block chain: Come up with a situation
where "the blockchain" wouldn't be a good fit.

~~~
eru
Oh, American sports metaphors. They fly right over my head.

A situation where a hyped up 'blockchain' wouldn't be a good fit: when the
organization is competent enough and/or the CEO has enough power to change
things.

Google probably falls under the former. Amazon might fall under the former and
definitely latter.

Let's quote Steve Yege to see why Amazon doesn't need "the blockchain" as an
excuse
([https://plus.google.com/+RipRowan/posts/eVeouesvaVX](https://plus.google.com/+RipRowan/posts/eVeouesvaVX)):

> So one day Jeff Bezos issued a mandate. He's doing that all the time, of
> course, and people scramble like ants being pounded with a rubber mallet
> whenever it happens. But on one occasion -- back around 2002 I think, plus
> or minus a year -- he issued a mandate that was so out there, so huge and
> eye-bulgingly ponderous, that it made all of his other mandates look like
> unsolicited peer bonuses.

> His Big Mandate went something along these lines:

> 1) All teams will henceforth expose their data and functionality through
> service interfaces.

> 2) Teams must communicate with each other through these interfaces.

> 3) There will be no other form of interprocess communication allowed: no
> direct linking, no direct reads of another team's data store, no shared-
> memory model, no back-doors whatsoever. The only communication allowed is
> via service interface calls over the network.

> 4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom
> protocols -- doesn't matter. Bezos doesn't care.

> 5) All service interfaces, without exception, must be designed from the
> ground up to be externalizable. That is to say, the team must plan and
> design to be able to expose the interface to developers in the outside
> world. No exceptions.

> 6) Anyone who doesn't do this will be fired.

> 7) Thank you; have a nice day!

------
polm23
I remember running into someone who worked at an insurance company in my home
state and mentioned their whole system ran on COBOL. I asked her how she hired
programmers and she explained they hired recent college grads with no
technical background and trained them from scratch, since it was easier and
cheaper than hiring experienced COBOL programmers. Looking back I'm surprised
they had the institutional knowledge to maintain systems that way...

~~~
StillBored
They don't. That is why they need to call back the guys that retired.

Any of the new hires with talent quickly discover they can double their salary
by doing pretty much anything else in IT and move on. IBM and their customers
want to turn the whole thing into a McDonalds style business, but the burgers
keep burning and they keep having to call people to come in and rewire the
microwave and unplug the toilet.

~~~
watwut
> Any of the new hires with talent quickly discover they can double their
> salary by doing pretty much anything else in IT and move on.

Not necessary. They do not know current frameworks and tools and probably
don't know the knowledge is transferable.

Effectively, they need to find a company willing to train them and the one
that hires by aptitude on whiteboard test only.

~~~
user5994461
Yes. Very likely that they can't get a better job. As the previous comment
said, they are trained from scratch having no experience in tech whatsoever
and mainframe is not remarketable to a web company.

~~~
eru
Google is willing to hire anybody who can solve a few basic algorithmic
problems.

~~~
user5994461
The hardest part of Google is to be accepted for an interview, which they
won't.

The second hardest is to actually answer the questions in the interview, at
that they don't stand a chance. Don't overestimate a guy who is doing a job
reorientation from a random degree, having barely touched a computer before.

Last but not least, Google is only in a handful of locations in the World,
banks are more common.

~~~
eru
The trick to get your interview at Google is to get somebody to refer you.
(And that's not that hard. If you can't find a Googler in person at a meetup,
hang out online and write to a few random Googlers. They get a hiring bonus,
so if you provide them with the information they need (it's a small
questionnaire about you they have to answer plus your CV), lots of them are
happy to refer you.)

The interview is a bit peculiar, and not all that correlated to what even
Googlers do on their day job, but it's trainable, and you can repeat every
year.

Do Facebook at the same cadence, but six month out of phase, and you got a
steady stream of interviews. Add eg Microsoft and some other companies, and
you are virtually guaranteed to land a decent job at some time. And that holds
even after I agree with your caution about overestimating a random but
intelligent person. It is quite the time commitment, though.

The locations weren't too much of a problem for me. But I do have to admit
that when I joined my preferred location of Singapore didn't really have any
engineers, so I settled for Sydney. And as your comment suggests, yes, I was
working for a bank in Singapore before.

~~~
user5994461
Yes, it's good advice, I know how this works. Meetups and linkedin messages.

You still need to be in a google location to come across a meetup (there are
only so few offices in the world) and your profile might need to be better
than "I dropped from art school last month. I like computers."

~~~
eru
I never finished my degree..

------
nickbauman
I've worked in the financial sector as recently as 2006. Without getting too
specific, some of the financial systems I worked on underpin large sectors of
the economy, across many banking chains. While Java was starting to take over
these systems, thankfully the banks and regulators were hedging their bets on
JEE with the knowledge that someone would eventually have to _take over_ these
systems.

They were starting to think about growing an intergenerational community and
started hosting applications in smaller chunks that individually comprised
less risk (long before microservices was a coined term) and thinking in terms
of APIS not implementations. They're not idiots.

Having said that it might be good to ask your local bank how much of their
systems are implemented in COBOL.

~~~
parthdesai
Anecdotal, but my friend knew VP of Information Technology at one of the big 4
banks in Canada really well. The VP told my friend to learn COBOL and if he
passed the interview, they give you a blank paper where you write down your
salary.

~~~
nickbauman
What this VP should have been doing was figuring out how to get out of that
scenario entirely.

~~~
parthdesai
Well VP could have been exaggerating a bit as well since we were uni students
back then and my friend was doing his internship there. He knew the VP through
robotics.

~~~
gpderetta
Note that, in a bank, VP is not a particularly important position. A senior
software developer anywhere else might have the VP title in a bank. It is very
possible that your firend's friend wasn't involved in any hiring decision at
all and it was just internal hearsay.

Source: I'm VP at a large bank.

~~~
parthdesai
Ah could be that as well, you know better. We were just young kids (still are)
and believed what he said.

------
skc
I work at a large bank in my country.

We have quite a few old gentlemen that work about four hours a day on insane
hourly rates doing this sort of work.

They seem to be living the life quite frankly, spending the rest of their day
boating and what not. It's like the perfect retirement for them in that they
get to stay challenged, get paid handsomely and still have most of the day to
actually kick back.

------
exabrial
15 years of bad executive decisions adds up to a problem? Oh man. Say it ain't
so.

------
creaghpatr
It's easy to forget that most businesses America aren't publicly traded
companies.

This guy found a profitable niche, realizes that it isn't sustainable long
term, but makes perfect sense for him and his peers. Good for him.

------
0898
There's a retail bank in London that still has its core standing order system
written in pounds, shillings and pence with a translation routine sitting on
top of it. It was supposed to have been rewritten as part of the millennium-
bug investigations, but it never happened.

------
JimmaDaRustla
The problem isn't COBOL developers, it's the Subject Matter Experts (SMEs) of
the applications themselves - developers don't know why something was built
the way it was, so maintaining and upgrading it is difficult from a logical
perspective, not a technical one.

When you provision a credit or debit card to an iPhone, it hits thousands of
lines of COBOL for my bank. I wrote the application that delegates requests
that come in on the payment network from Apple. Our issue isn't finding
developers to maintain it, but if the application sits stagnant for decades,
then the problem exists that we don't have product expertise to do the work.

------
catmanjan
From the article "Experienced COBOL programmers can earn more than $100 an
hour"

In Australia that's a lot, but my understanding is that it's common for
American developers to be on way more than this...

~~~
cdsx
I'm curious what your experience in Australia has been for you to think that.
In my experience that's not what I'd call a lot in Australia, most commonly
your average basic IT contactor will charge $80-120/hr, let alone any sort of
speciality.

~~~
bananaboy
What's a basic IT contractor? $100 seems to start getting to the upper end of
what I'd call a basic IT contractor. > $100 seems to be for more
management/architect type roles according to
[http://au.hudson.com/portals/au/documents/salary%20guides/20...](http://au.hudson.com/portals/au/documents/salary%20guides/2017/HudsonSalaryGuide_TechnologyDigital_AU.pdf)
and
[http://www.greythorn.com.au/media/greythorn/greythorn%20mark...](http://www.greythorn.com.au/media/greythorn/greythorn%20market%20insights%20and%20salary%20guide%202014_2015%20australia%20pay%20rate.pdf).

~~~
lacampbell
Don't most contractors use recruiting companies when they start out, then rely
on word of mouth/networks? Might be why the hudson/greythorn rates seem low.

~~~
bananaboy
Hm, yeah that could be true. I think the big organisations like the banks,
insurance companies, super companies, Telstra, NBN etc don't often take on
contractors directly either.

------
mbloom1915
It's certainly not just banks, very much utilities too. putting off
investments in core IT capabilities has become the norm and when you've dug a
hole this deep, might as well keep on digging!!

------
ltratt
My gut feeling is that one can minimally maintain systems in "forgotten"
languages indefinitely if they're basically stable and feature complete. But
it seems that people come unstuck for systems that still need to evolve --
even if that evolution is slow and/or sporadic. The obvious alternative is to
rewrite systems in new languages, but that is generally prohibitively
expensive. Even for those that can afford the expense, the new system tends to
not to take considerable time to bed in, so flipping the switch from the old
to the new is only for the brave or those with excellent PR departments.

Although we didn't have this in mind when we started on this work, I've come
to think that language composition might offer a way out of this mess (I
immodestly offer [1] as an early example). If one composes an old and a new
language, one can migrate a system from an old to a new language bit-by-bit.
As well as amortising the translation cost, it also means that one can
minimise the "switch flipping" problem: disasters seem much less likely if the
composed system can be introduced gradually.

[1]
[http://tratt.net/laurie/blog/entries/fine_grained_language_c...](http://tratt.net/laurie/blog/entries/fine_grained_language_composition.html)

------
unfocused
I dealt with something similar when I was a programmer. We had inherited a PDF
publishing module written in 3B2 from the late 90s. (Google that language and
you won't find much. Wiki article here:
[https://en.wikipedia.org/wiki/Arbortext_Advanced_Print_Publi...](https://en.wikipedia.org/wiki/Arbortext_Advanced_Print_Publisher))

The most I've personally seen a contract was for hiring a consultant to help
fix bugs in this 3B2 language that nobody knew. The cost for bug fixing and
onsite training was $5000 CDN, per day. That was back in 2010. The day was 7.5
hours so this guy's rate was $666/hour (Now that I look back, maybe that rate
was an inside joke or something).

We ended up hiring him for 5 days ($25K contract) as that was all that was
needed to keep this module going, and we crammed a bunch of programmers in a
room with him to see if we could learn a thing or 2 - the end result was
negative. Nobody could learn it because we just didn't want to touch it.

It took another 5 more years to kill off that module and move to an open
source solution. Now before anyone does any straight math do see how much this
consultant made per year, you have to understand that work was very limited,
and eventually, everybody wanted this language and publishing software to die
off.

------
coleca
That's why I'm keeping my COBOL for Dummies and MVS/JCL handbooks as my
retirement plan should the markets go south on my 401k. :-)

~~~
teh_klev
/digs out old Data General Interactive Cobol manuals :)

~~~
defined
DG? You're really giving away your age ;)

------
derrickdirge
Boy this sounds familiar.

It's not just banks; many state (and presumably federal) angencies have this
exact problem.

The agency I work for now has a large COBOL code base that's been in the
process of being converted to Java for a very long time now with varying
degrees of success. I frequently have to read through COBOL to figure out what
a particular project was supposed to do in the first place. It's not hard to
understand, but it's certainly not immediately intuitive for someone who came
up on Java.

We have the privilege of still having a few of the authors of the original
systems hanging around, but we won't for much longer.

------
killjoywashere
A couple of thoughts:

If COBOL is such a big deal, a bank would get a steal of a deal if they funded
some university chairs for professors who are willing to teach COBOL. "High
Stakes Financial Software Engineering 332".

Others are indicating that the banks would rather hire a novice and let them
learn the system. Seems like they have some sort of long term onramp
procedures. Or their current crop of almost retirees are harboring a deep-
seated desire to see the world financial system go down in flames. Which would
be odd for someone who will presumably be depending on a financial instrument
like a pension or 401k for income.

------
y3sh
Not just banks, but large healthcare companies too. This is Accenture's bread
and butter -- charging $120+/hr for analysts to reverse engineer COBOL into
rules engines, then outsourcing a J2EE implementation. Last time I checked a
large healthcare provider was paying over a million/month to maintain a
mainframe claim processing system.

------
Kiro
> Experienced COBOL programmers can earn more than $100 an hour

Sounds low? That's what I'm charging as a web dev consultant.

------
xacaxulu
"Experienced COBOL programmers can earn more than $100 an hour when they get
called in to patch up glitches, rewrite coding manuals or make new systems
work with old."

$100/hr doesn't seem like a good rate for a true 'niche' when $185-$250/hr is
going around for everything from Hadoop to Kubernetes.

------
doggydogs94
Just went into Autozone and the interface looked like a IBM CICS screen
scraper. Went to my insurance agent; we looked at nice screens, but when real
money became involved, we popped into this ugly legacy screen.

~~~
grogenaut
Or maybe it's because keyboard input is faster for a lot of heavily repeated
form type things than a web interface.

Web is nice when users don't know how to use it. When they do they're not
really looking 1/2 the time.

~~~
jlebrech
terminal screens needn't be old. they can be quite fast to operate.

------
surgeon7
Interesting how COBOL still stays relevant till this day. "They don't make 'em
like they used to" is a cliche more commonly heard for hardware rather than of
software.

------
rmc
> _Experienced COBOL programmers can earn more than $100 an hour_

Only $100 ph??

~~~
jlebrech
yeah but you're holed up 7 days straight to fix something, so it quickly adds
up.

------
LurkerAbove
I had a recruiter contact me about a COBOL --> Java project a couple of weeks
ago. I don't remember the $100/hour part though, seems like that would've
stuck with me!

------
kazinator
_" The call back to the bank was something of a personal vindication for me,"
he said._

Being called back because nobody can figure out the undocumented hairball you
left behind is a vindication?

More like, your plan to ensure endless job security, having appeared to have
faltered, was vindicated.

 _You_ being vindicated means that someone admits that you should not have
been let go; it was a mistake based on misinformation or whatever.

------
bertlequant
"Come check out our data center". Tape robots. Compliance's friends.

------
nraynaud
dear Cowboys, start your ride in the morning, the towns are specifically
spaced such that you can reach the next shelter in one day, avoiding the
dangers of riding during the night.

------
douche
Businesses in general seem to have a really hard time dealing with training
and institutional knowledge transfer these days.

As an example from another industry, my father was working as a
mechanic/greaser in a power plant for the past 17 years, alongside a
millwright who had been there for 30, since they opened the doors. In the last
six months, they both retired - after telling management what their timetables
were about five years ago. The company never brought in anyone to train under
them and learn all the minutia of how the plant works that is locked away in
their heads. They haven't even hired anyone full-time to replace them.
Needless to say, equipment all over the plant is starting to break down,
because simple things like greasing bearings regularly isn't being done
properly, and when things break down, they aren't being put back together
properly, because no one knows the idiosyncrasies of the equipment. Every hour
this plant is down costs tens of thousands of dollars.

The millwright is starting to get called in as a consultant for three times
his previous hourly wage.

~~~
technofiend
>The millwright is starting to get called in as a consultant for three times
his previous hourly wage.

Sounds like he should have retired ten years ago. :-) :-)

