
Aging Canadian government computer systems at risk of 'critical failure' - ChrisArchitect
https://www.cbc.ca/news/politics/federal-it-systems-critical-failure-1.5448871
======
latch
About 15 years ago, I became a full time employee of the Canadian Government.
I was part of a team that was rebuilding a pretty huge and critical system. It
was a disaster. I left after a year. The gun registry before it and the
Phoenix pay system since, and many other failures in between..it seems pretty
hopeless.

I initially put all of the blame on the expensive consultants, that were doing
most of the work, for being technically awful (writing software like Fowler's
Analysis Pattern book was a holy grail).

While I still think they bear the brunt of the blame, in retrospect everything
was awful. The project was both a business and technical transformation at
once. There were too many managers, too many business experts and overall poor
leadership. Rather than breaking it down into manageable chunks, it just kept
growing and growing. There was no concern or accountability for waste.

Sometimes replacing a legacy system with a new system is difficult. But most
of the time, it can be done as a slow and steady transition...so long as egos
and ambitions stay out of it.

~~~
refurb
Canadian here, but left 20 years ago, but I read CBC news daily.

That Phoenix pay system was a total cluster that lasted for years. I remember
reading headlines and thinking “they haven’t fix it yet?”.

Made the ACA website in the US look like a resounding success. People were
literally not getting paid for months (years?) and the gov’ts was like
“working on it!”.

~~~
Harvey-Specter
They still haven't fixed it. My partner works for the federal government. She
was bumped up to a new pay scale and suddenly stopped being paid for 2 months.
When she did start getting paid it was at the old level and she hasn't
received the missing 2 months salary yet. She had to open three separate cases
to 1) receive the missing pay, 2) have her pay scale updated in the system,
and 3) receive the difference between her old salary and new salary for
however long they continue to pay her at the old rate.

One of her coworkers hasn't been paid in almost 6 months.

~~~
refurb
Nuts! You’d think after a few weeks they’d hire a ton of folks and do it by
paper until it could be fixed.

How many people can go months without a paycheck.

------
owroomexorcist
I work on one of the systems called out in the article. It's a fabulous
disaster. I don't think it'll keel over and die anytime soon, but it
absolutely should be phased out.

The problem is these big projects, like modernizing core social service
systems, are given big budgets and big expectations. Then they start a big
team to work on it, with short deadlines and not enough direction. At some
point it is discovered that the legacy systems business logic is mostly
undocumented/baked into a mainframe/functioning through sheer luck/nothing but
PL/SQL and DB triggers. Then the several contacting companies are brought in
to try and sort out the mess, solutions are purchased from billion dollar
private companies (all of which have screwed over the Government of Canada one
way or another), and project management is shuffled around as fast as HR can
fill out the forms. After three years the money runs out, the project dies,
and then we try again in two years.

The solution is never a slow and steady modernization. It's never small,
focused teams working on improving or rebuilding individual components. It's
never upgrading infrastructure, or moving to modern best practices.

It's always all or nothing. Solve the problems, without fixing them.

~~~
zzzzzzzza
whats wrong with pl/sql... and what do you think modernization entails?

~~~
owroomexorcist
Another commenter summed up my feelings on PL/SQL. It's not that it's
inherently bad, it's the bad practices that seem to follow.

Modernization to me (specifically in the context of the GC) means
refactoring/rewriting systems to follow modern industry standards and
practices. Things like proper version control, automated testing,
automated/semi-automated deployments, and monitoring and logging.

I think there are major benefits in adopting these practices for both the
Canadian public and the developers building/maintaining the solutions.
Unfortunately, getting a budget to move the source code from a shared network
drive to Git is next to impossible. And God forbid you want to spend time
adding any sort of testing.

------
fatbird
I work for a digital services agency currently working with several provincial
governments, and have some perspective on this. I call it "the IBM hangover".

At some point, someone gives IBM or various consultants millions of dollars to
implement some IBM software--Powerbuilder, Java, Domino, etc.--which they do,
following RUP, which means a pile of UML docs on top of it. It actually works,
the gov't says "great, we can stop funding ongoing dev for a couple decades."

Two decades later, CGI has taken over the IBM project for half the maintenance
fees because they know how to do RUP, and thus meet the piles of NFRs IBM left
behind (such as "all metadata changes must be auditable", meaning all UML
diagrams must have a documented review/approval chain separate from source
control").

I sat in a meeting where a stakeholder showed us the report generation page,
and said "if we click on this link, it crashes the server, and we have to open
a ticket to restart it".

Software does rot, and gov't procurement processes and requirements are those
left behind by IBM to prevent smaller, "agile" agencies from doing meaningful
work. To my mind, there are two flaws here: one is letting IBM in, in the
first place, to set the ongoing standard; two is refusing to acknowledge that
every organization needs an organic software dev capability that can use
outsourcing as a resource multiplier, not as an IT replacement.

------
vaughnegut
Does anyone have any idea what outdated technologies they're using (notably
the the ones that are sixty years old)? It would be interesting to learn which
software has managed to stick around for so long and what it does.

Two anecdotes:

1) I know someone who deals with legal documents for the Canadian federal
government and she has multiple word processors (of various versions) for
these documents. Since minor changes can affect the meaning of these
agreements, they are only edited or amended in the software in which they were
originally created. She is understandably very excited for when the ones
created in old versions of Word Perfect expire and a new agreement is created
in newer software.

2) Only in 2019 did the US Nuclear Command transition away from their own 50
year old technology: floppy discs from the '70s. [1]

It always fascinates me to see what sticks around, for what reasons, and how
it affects people's work. I've heard stories of developers creating fancy UIs
to cover up ancient Fortran software, so that it's less painful to work with,
but they don't need to replace the underlying software.

[1] [https://www.nytimes.com/2019/10/24/us/nuclear-weapons-
floppy...](https://www.nytimes.com/2019/10/24/us/nuclear-weapons-floppy-
disks.html)

~~~
skissane
> 2) Only in 2019 did the US Nuclear Command transition away from their own 50
> year old technology: floppy discs from the '70s. [1]

From my reading of the article, it appears they are still using 1970s era IBM
Series/1 minicomputers. It sounds like all they've done, is replaced the
physical floppy drive with a floppy emulator. This is a device which attaches
to the legacy IO bus, and appears as a floppy disk drive to the minicomputer,
but the actual data is stored in flash memory instead of magnetic media.

[https://en.wikipedia.org/wiki/Floppy_disk_hardware_emulator](https://en.wikipedia.org/wiki/Floppy_disk_hardware_emulator)

~~~
Spooky23
You need to think of things like that like a low-scale industrial system. It
needs to do what it does.

If you go to a candy-store at the beach and see a 150 year old saltwater taffy
machine, you say "ooh, cool". The thing that makes a missile go up is
essentially the same thing, it needs perform it's function to spec, period.

~~~
dsfyu404ed
Yup. In tons of industrial settings it's common to have machines that are a
century or more old.

I know someone who works for an industrial gear company and routinely
reconditions things like draw bridge mechanisms or a rubber rolling mill (do
they call it a mill?) and most customers just want things to keep on working
like they've always been working since they've built their process around them
and machine throughput is rarely the bottle neck. The improvements customers
go for are modern bearings (quiet and cheaply serviceable) and if they have to
have gears made they generally opt for a more modern tooth profile
(potentially stronger, quieter and more efficient).

------
geocrasher
On a much different scale, I ran into a similar problem once. Not for me, but
for a customer. About 10 years ago I did a PC breakfix job onsite at a small
business that made trophies. Their main business was bowling leagues and other
local sports. The computer I was there to fix was fine. But the computer that
ran the engraving machine- the one that they made all their money with- was an
IBM PC/XT from the early 1980's. No hard drive, everything was on 5.25"
floppies.

I mentioned that they should replace that thing sooner rather than later, but
their objection made sense for their use case: The replacement would be many
thousands of dollars and this old setup _still worked_. The computer had one
job, and it did it. Who knows, maybe it's still there, still mashing away on
its floppies every day. It wouldn't surprise me. If it is, it's being held
together by nicotine deposits and luck.

~~~
choonway
When the time comes, they could just order a cheap machine from China to
replace it.

~~~
geocrasher
Indeed. At the time, that wouldn't have been an option. And, given their
technical abilities (I was there to fix a _really_ minor problem) I doubt they
could piece together something that your average maker-type such as us could.

------
jariel
IT might kill government. It has just the right mix of difficult to predict,
measure and evaluate technical competency etc. wherein a small project can
blow up to $1 Billion dollars.

This is where Western Governments are actually corrupt, but it doesn't show up
in Corruption Transparency Index.

I'm not ideologically a 'small government' person, but I have absolutely no
faith in our government's ability to do anything reasonable in IT.

Sometimes I think we need 'government in a box' IT solutions. Sadly, even if
we did, they'd still labour over them in some way and make it expensive: the
whole point is for vast cadres of the civil services, and consultant/lawyers
etc. to suck money out of the system.

~~~
sgift
You hear this attitude all the time on HN, but never see any kind of
disruption happening. After working with big companies (in many ways not so
different from governments) and also a few government agencies I feel like
"this is all corrupt, so we cannot do anything" is mostly a cop out, so people
don't have to accept that maybe they were a bit naive when they ran around
"this is all needlessly complicated, it could all be so much better and more
simple!" and maybe the complexity is there for a reason.

~~~
jariel
No. The Canadian government spent $1B on a simple gun registry, a basic CRUD
app that barely worked. US 'healthcare.gov' doesn't need to be that
complicated, but it was $2B, screwed up by a Canadian contractor, CGI. A small
team of Google devs had to come in and fix it.

Most companies that screw things up that bad, will fail. If they don't, it's
still their right to waste money on dumb projects - it's their money.

Government failures (at least in Canada) generally exceed those in the private
sector for that reason, exposing the dire systematic problem if 'no
competition, no oversight, lack of competency' on a scale rarely seen
elsewhere.

Not only is there 'no incentive' to fix problems, often there's also a
negative incentive.

It's 2020. The technology to put my medical history online has been available
for 20 years. Ontario, Quebec etc. have still completely failed to do this. I
still have no easy way of finding out which clinics are available for me, and
when I do go to a new one, they have to open an entirely new file, totally
unaware of my historical medical issues. To make matters worse, it's literally
illegal for me to pay anyone to provide me with medical services. It's
kafkaesque.

A very basic medical history system, that merely documented doctors notes etc.
could be done 'on the cheap' (relatively speaking) - but it's far from
happening.

Even an intelligent regulatory mandate could solve the medical records issue,
i.e. providers must participate in XYZ system, with ABC components, designated
by the government. But we can't even have that.

It's really bad and I don't see any path to getting better until government
develops a whole new attitude towards IT.

~~~
Spooky23
Replace government with large, complex organization. There is litle
difference. I've worked in government and large companies, and they are pretty
much the same.

The only difference is the governance structure. In .gov, you tend to have
professional / civil service people at the senior director level who know
their business inside and out, with a political layer of management who drive
change and vary in competence.

Medical records are a great example of how .gov/.com doesn't matter. When
stuff gets complex, IT sucks.

~~~
jariel
"Replace government with large, complex organization. There is litle
difference. I've worked in government and large companies, and they are pretty
much the same."

I don't agree at all.

Big corporations can fail to do many things where it's not really important,
so it might seem like 'failure' but mostly it's a function of market
conditions. Other big failures (say Boeing 777) are understandable due to
complexity.

Very few groups on earth can build such airplanes.

Anyone can build a gun registry.

I loathe how long it takes my bank teller to speak to me, but my banking
services are in the end, amazingly cheap for what they provide.

Governments do a reasonable job at things like contract allocation for road
maintenance, some kinds of construction, but they generally do a bad job
operationalizing anything.

------
ncmncm
The way businesses solve this, just about every time, is to run an emulator of
the old system on current hardware.

I have heard of systems with five layers of emulation, yet still quite a lot
faster than the original machine.

I gather Bloomberg has begun to do this, emulating SPARC user-space on Intel
so they can stop paying Oracle to support long-since EOL'd 32-bit equipment
and OS. (Or maybe some sort of hybrid.)

I can understand why no expensive consultant would suggest it; it is very
cheap, and quick to set up. No fat contracts there.

------
animex
I wish we'd develop a similar agency to the US Digital Service. I think any
modern government should have a digital/IT branch... a MIO (Minster of
Informatics) etc. I know it's idealistic but I think we could pragmatically
solve Canada's digital issues carefully without costing us billions in
consultant-overloaded failures.

~~~
frosted-flakes
There's the Canadian Digital Service, modelled after the UK's Government
Digital Service (GDS). Provinces like Ontario are also creating their own
Digital Services groups, and Ontario's online services have improved
tremendously because of it.

[https://digital.canada.ca](https://digital.canada.ca)

[https://www.ontario.ca/page/ontario-digital-
service](https://www.ontario.ca/page/ontario-digital-service)

~~~
dleslie
The pay is _terrible_ for the skills required; I interviewed some time ago and
would've taken an 50% cut in compensation to work there.

~~~
frosted-flakes
That's good to know. I was/am interested in working there. I might apply
anyway just for the experience (currently not working in software development,
but I'd like to get back into it).

------
aurizon
Consultants in Canada are usually friends of the government in power and their
contracts are usually narrowly specified to make the selection process
converge to their group. We had a useless proprietary educational computer
system in the 80's and 90's that failed The ICON - note the last three letters
of the name!!!. They appeared visionary to the public in press releases, but
they were proprietary Kludges that endured, and froze their state of
technology while the field raced ahead. They never
worked.[https://en.wikipedia.org/wiki/ICON_(microcomputer)](https://en.wikipedia.org/wiki/ICON_\(microcomputer\))
But the consultants were all over it, like ticks on a steer, they grew fat.
The Ontario Archives refused to accept the donation of hardware and SW for
saving = dead as a doornail

------
diminish
I recently got a visa to Canada. The J2EE applications for the visa
applications didn't work for a full weekend. I've opened some tickets, and
found cookie-clearing as a fix for some issues.

Finally went there for a sofware event and people in Canada are sweet, calm,
happy and non-stressed.

------
908B64B197
I feel that one of the issues that's never talked about is how government
contracts are awarded. 'Analysts' and 'Architect' prepare a document outlining
the spec of the program. Once the document is out, every consulting firm is
free to access it and place a bid on how much they would charge to implement
what was outlined in the procurement document. The government is then forced
to pick the lowest bidder. It doesn't matter if what's asked by the government
doesn't make any sense, the engineers on the contractor's side are forbidden
from contacting government employees (for anti-corruption reasons).

I've heard many stories where the contractor knew the job would have to be
done twice the moment they read the procurement documents. But they couldn't
voice their concerns. And if they suggested doing what they knew was the right
thing that would have made them ineligible for the contract as it wasn't what
was required. Future-proofing the bid or trying to deliver something closer to
what they ended-up shipping was also not possible because this would have made
them more expensive than the bidders following exactly the request. In the end
they ended-up rewriting most of the code at their usual billing rate on top of
the original fixed cost contract.

In the case of Phoenix, I've read a lot of media articles outlining how bad of
a job IBM did but despite all this it seems the contract itself was never
challenged in court. What I heard from internal sources is that IBM did ship
correctly all what was asked for in the contract, it's just that the
government workers drafting the requirements didn't understand their own
payroll needs enough to properly articulate them.

Of course CBC, the state-funded media where everyone is on the government's
payroll, won't outright blame their bosses. But they would get sued and lose
pretty bad in court if they claimed the contractors didn't deliver what was in
the contract. So you get articles with a weird spin where they try to blame
the contractors without going too far and paint the government as a victim.

~~~
moltar
I also read that the project was pushed live prematurely for political
reasons, despite the warnings from IBM that it wasn’t production ready.

------
Humdeee
I believe that in Ottawa, there's a negative stigma with those who are in tech
and part of government because of so many of these systems. Although it's sad
to see, it's not completely unwarranted. I'm about to get an early start on my
taxes and Canada's tax website is a slug and spits out errors, timeouts, etc.
It's been like this for years. Pressing refresh for the 4th time and it
manages to load the page I want slowly becomes the tech variant of Stockholm
Syndrome. I am biased, yes, and that comes from countless frustrations dealing
with government tech.

Years ago, I had a past employer that would immediately toss resumes from
applicants that had recent employment with the government. Yikes.

------
bregma
Thankfully non-government industry is exempt from this kind of thing. For
example, the airline ticketing industry has kept up with modern technology and
paradigms with its corruption-free SABRE system. Banks these days exchange
trillions daily through modern full-stack phone apps.

~~~
winrid
That full-stack phone app probably puts an item on a queue for a COBOL
mainframe :)

~~~
buran77
This is for the old timer banks, the ones that had their HQs built around that
mainframe. Take newer fintech banks and you're in for a treat. Some of the
backends are so janky it's only a matter of time before one screws up badly.
Most of the effort goes into the app-polish because that's the visible part
that makes it into the media.

~~~
moltar
Which fintechs have their own banking system? I thought they were all riding
on someone else’s banking license and infra.

