
Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B - RmDen
http://www.itprotoday.com/hybrid-cloud/canada-scrap-ibm-payroll-plan-gone-awry-costing-c1-billion
======
dctoedt
Reminiscent of the state of Indiana's lawsuit against IBM for allegedly
botching a project to automate the state's welfare system — lots of finger-
pointing on both sides; the trial court's 65-page decision started out with
the words, "Neither party deserves to win this case" [0]. The case has been up
to the Indiana Supreme Court already [1]; on remand last summer, the trial
court found that IBM is liable for USD $128 million [2].

[0] [https://www.scribd.com/document/100544020/Indiana-IBM-
Decisi...](https://www.scribd.com/document/100544020/Indiana-IBM-Decision)

[1]
[http://www.in.gov/judiciary/opinions/pdf/03221601shd.pdf](http://www.in.gov/judiciary/opinions/pdf/03221601shd.pdf)

[2] [https://www.indystar.com/story/news/2017/08/07/court-ibm-
owe...](https://www.indystar.com/story/news/2017/08/07/court-ibm-owes-
state-128-m-welfare-privatization-dispute/544789001/)

~~~
dpweb
Sales and Engineering - very different competencies. Companies like IBM are
NOT technology companies. They are sales-culture oriented, and their product
HAPPENS TO BE technology. In fact you could argue, the only way to win big
enterprise contracts in the first place, is to be a sales-culture company.

But after selling the deal, their workers have to solve a difficult
engineering and organizational management problem simultaneously (the
project).

They use contractors (devs) which proves they are not an engineering but sales
culture. So, you need to bridge the two worlds. That's the PMs. These are the
most incompetent members of the team. The devs very often are very smart.

Draw horizontal lines on the org chart. The sales guys are at the top. They
are generally competent. Proof = they sold a massive deal. The devs are at the
bottom. They are often competent. They have to be or they wouldn't get hired
or find work. You can PROVE someone doesn't know dev work. But, they don't
have enough POWER to change things or fix things.

Reasons for failure:

1\. The middle. This is the breakdown. Many PMs often have no real skills.
ORGANIZING for success, given a complex technical and organizational problem.
2\. The projects are too big. Any huge project is from the get-go at an
unacceptable level of risk. Decentralization, Deconstruction is powerful. The
projects must be broken into smaller pieces to be managed.

~~~
kev009
I disagree, IBM is a multifaceted company.

These kinds of projects fall under what used to be called IBM Global Services
[https://en.wikipedia.org/wiki/IBM_Global_Services](https://en.wikipedia.org/wiki/IBM_Global_Services).
It could be compared somewhat to EDS (HP), Accenture, Perot Systems (Dell)
etc. I've never heard of over-delivery from any of these kinds of outsourcing
arrangements, they always seem so obviously destined for boondoggle.

IBM proper, the one that makes mainframes and POWER and DB2 and a ton of
operating systems and storage etc is very much a technology company. Some of
their best products have the worst sales and marketing efforts. I'm working
directly with the senior leadership of the POWER group right now and there are
no salesmen in sight.. the technology will either sell itself or not. When we
met in person the first time the GM told me "we can build any kind of computer
you want" \- meaning microarchitecture changes, SERDES configuration, new
board layout, sheet metal, OS, application tweaks. Not a lot of companies can
do that. There is hubris, less technology, and lack of technical value at FANG
or most startups or whatever your benchmark is in comparison.

IBM Research is one of the only remaining great industrial research
organizations
[https://en.wikipedia.org/wiki/IBM_Research](https://en.wikipedia.org/wiki/IBM_Research).
Their results speak for themselves.

~~~
stochastic_monk
I’m surprised the Stratix FPGA platform isn’t getting more marketing. Half a
TiB/s memory bandwidth [0] (white paper) when GPU’s memory transfer is the
bulk of the overhead could help Intel make up for lost gains in SIMD
application marketshare.

[0]: [https://www.altera.com/content/dam/altera-
www/global/en_US/p...](https://www.altera.com/content/dam/altera-
www/global/en_US/pdfs/literature/wp/wp-01264-stratix10mx-devices-solve-memory-
bandwidth-challenge.pdf)

~~~
kev009
Maybe misparented comment?

HP has GenZ, IBM is going to move from DDR or DDR buffers to CAPI attached RAM
-- expect to see HBM2 attached to CPUs in 2019. I'd be happy to discuss that
kind of thing in email.

~~~
stochastic_monk
Certainly misplaced! It was late, and somehow I scrambled IBM Research and
Intel. I was thinking about how Intel's marketing is generally quite bad,
whether it's unfair benchmarks or not knowing which products to actually push.

Please disregard.

------
watertom
I was the CTO for a large, Fortune 100 company.

When I first took over the position I would regularly get requirements
documents for internal projects that were 3 or 400 hundred pages. The project
team would dutifully carry out the requirements gathering process, everything
was meticulously documented, with data flows, and process maps, etc.
Everything was a requirement, everything was mandatory, and everything had top
priority. IT wasn’t allowed to say no, wasn’t allowed to criticize the
business, and wasn’t allowed to analyze if what was specified made sense, or
would work. Luckily I had a CIO that was willing to back me up when I started
rejecting these projects. In six months I killed 10 or so very large multi-
year software projects, in each case the business was forced to use existing
software and change their processes. We saved a ton of money and implemented
the solutions in weeks not years. These would have projects that had upwards
of 100 people for 2, 3 or 4 years, all because the business wanted a “perfect”
process.

I consulted to the DoD and this pattern was repeated, over and over again.

Most assuredly this is what happened with the Canadian Payroll project. Every
person had their pet requirement(s) and they made sure they “got them in”, in
the end the system that was specified couldn’t be built.

It’s like SAP, if you buy SAP take it out of the box and adapt your processes
to SAP, if you do you’ll have great experience. If you try to customize SAP,
it going to get very expensive, you’ll have a ton of problems, you won’t be
able to take upgrades and hou’ll Need a ton of staff just go try to keep it
running day in and day out.

~~~
canada_dry
I've participated on several >$10M Canadian Gov't IT projects over the years
and you nailed it.

Business groups will fight tooth-and-nail to keep their existing processes -
partly to maintain status-quo, but also because the folks who represent the
business know little about system's analysis/design, so they just simply
aren't in a mindset to re-engineer.

Combine this with the vendor's (not so hidden) agenda of milking the gov't and
it's a recipe for disaster... every damn time.

~~~
3pt14159
I've consulted to the Canadian Government on both a data science project and
on cyber security as well. The technical people inside GC know what they need
and it isn't IBM building custom software for something like payroll. They
should use off the shelf stuff for the 95% of employees that have run of the
mill circumstances and just use humans to handle the super long tail of
bespoke needs that no other Canadian employer has to deal with.

The problem is that the political side of the government doesn't trust their
own technical staff and they get convinced by these horrible consulting
companies like IBM that the only way to write good software is to spend
hundreds of millions on it but don't worry it will pay for itself with all the
cost savings on staff.

They try to minimize risk through bureaucracy but they end up getting the
opposite of what they want. Less useable, less secure software at 50x the
price. What they would do if they understood software is communicate to the
public about how good software projects need fast iteration and that mistakes
might be made sometimes, but that the important thing is that they are easy
and fast to fix and that there are risk mitigation strategies to stop mass
leaks when penetrators get into networks.

Also, the CSE should just be empowered to take GC servers offline whenever
they don't conform to a predefined list of security practices. It's 2018 the
fact that all of GC isn't on HTTPS / HSTS is quite frustrating and it requires
all departments to get on HSTS preload lists because of the stupid way gc.ca
is a subdomain of .ca.

------
goodroot
As a Canadian who has worked at IBM, I'm not surprised that this didn't find
success. In IBMs defense, I don't believe our public sector has the experience
or aptitude required to act as a supporting interface for a job of this scale.

With that being said, IBM isn't a successful technology company with a proven
record building good software products. They were an unwise choice from the
get-go. IBM is a successful financial engineering and sales firm that sells
things, then scrambles to hire/acquires to get things done to an ever-
evolving, always justifiably rough spec. They quack and then start the hack-
show.

What do you expect from the organization that invented "FUD"
([https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt](https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt))
as a way to emotionally manipulate perspective clients into making a
purchasing decision?

~~~
djrogers
> IBM isn't a successful technology company with a proven record building good
> products.

Whoa, what? Sure, some of their achievements are in their past, but that claim
really overlooks a _ton_ of history.

~~~
goodroot
You're right. That was too sweeping, in fairness. I've edited in 'good
software products'; that record is categorically abysmal. IBM makes amazing
mainframes and physical technical systems, and conducts some fascinating
cutting-edge research, and even built really effective punch-card machines for
use within Nazi concentration camps.
([https://en.wikipedia.org/wiki/IBM_and_the_Holocaust](https://en.wikipedia.org/wiki/IBM_and_the_Holocaust))

~~~
ugh123
You're mistakenly conflating IBM's consulting business with the rest of their
current technology, both hardware and software.

~~~
flukus
No, IBM has conflated their consulting service with the rest of their
technology by running under the same brand name.

If they want the benefits of sharing that brand between operations then they
deserve the negatives that come along with it.

~~~
aisofteng
No, IBM has not done that. IBM Global Business Services and IBM Global
Technical are their own business units.

------
zaroth
Not having ever worked on anything like this, the death knell for these
projects must be the inability to ever reduce functionality into a reasonable
core set. Basically the 80/20 rule but where there are practically an infinite
number of edge cases that stretch the project on indefinitely?

You could serve 80% of the payroll burden with software that cost $10m. 90%
coverage costs $100m. 98% costs $1,000m. And 100% costs $∞

~~~
mseebach
> You could serve 80% of the payroll burden with software that cost $10m. 90%
> coverage costs $100m. 98% costs $1,000m. And 100% costs $∞

I read something in a trade journal probably probably about 15 years ago, that
really stuck with me: When businesses buy big software solutions (I think the
article was about ERP systems), they should prepare to adjust and change their
processes (within reason, of course) and not just insist that the software be
modified endlessly to suit every little ossified process and preference. Not
just because it's very expensive, but because the resulting system will be
brittle and incompatible and taking advantage of further developments.

Too many buyers of software would benefit immensely from taking a good, hard
look at those remaining 20% and making some hard decisions about what is
actually important. Sure, for a government payroll system, there is limited
room for movement, but I'd suspect that the majority of the long tail are
people on weird, custom contracts and the correct solution is actually to hire
a room full of clerks to handle these in Excel - and/or applying political
pressure on the hiring parties to get their contracts adjusted to fit standard
payroll schemes.

~~~
greenshackle3
Having worked on ERP software for medium businesses, that sounds spot-on to
me; in my experience the biggest challenge was not so much technical as
getting clients to change their processes so they can be systematized in a
sane way.

There is a lot of resistance from employees not willing to change the way they
work. Even seemingly simple things like standardizing language / glossary
between departments so we can have a single representation in the system can
be an uphill battle.

~~~
adrianratnapala
As an employee of organisations I can tell you one reason for resistance: this
sort of systematization ends up being endless meaningless churn.

------
jellicle
Incidentally, the error was here:

"The project was meant to save costs by firing 1,200 employees handling
payroll at various departments around the country and replacing them with
about 500 people in a centralized location using Phoenix to handle most of the
government’s payroll needs."

It's not software problems, or not primarily software problems anyway. They
fired EVERYONE who knew anything about payroll in the entire Canadian
government, all at once, and hired 500 brand new people in one call center to
work with the brand new software as it rolled out. The new software doesn't
include code for a lot of "special cases" which together represent a lot of
cases. The new people have no idea how to solve most pay problems (people
problems OR software problems). They get four days of training.

[http://www.cbc.ca/news/canada/ottawa/phoenix-falling-pay-
cen...](http://www.cbc.ca/news/canada/ottawa/phoenix-falling-pay-centre-
miramichi-1.3687476)

"Phoenix was supposed to cut our work in half. It actually doubled our work.
So, you can't get rid of 2,000 people ... and double the work and expect the
work to get done," one said."

Overall, even if the software worked _perfectly_ the execution would have been
a disaster.

------
kabdib
Was not surprised to see Oracle and PeopleSoft involved.

I hope they get the crap sued out of them. That stuff is toxic, and I pity any
organization that Oracle got its claws into using that junk.

~~~
xienze
> That stuff is toxic, and I pity any organization that Oracle got its claws
> into using that junk.

In my experience, no one is really duped by these big companies. The people
writing the checks generally start their search with magic quadrant leaders
and then further narrow it down to the biggest players because "we can't trust
our business to some no-name." It's really a situation that companies
willingly put themselves in.

~~~
giobox
I've yet to see any way to procure custom "Enterprise" software that isn't
awful, and sadly I do believe duping is a huge part of it.

The duping starts long before the Oracles/IBMs/Whomever of the world ever
reach a customer too - the "analyst" firm role is a huge problem with
companies like Gartner/Forester who dream up the "Magic Quadrant" that people
purchasing this stuff end up stuck relying on. I've seen in several roles the
enormous gulf between what Gartner/Forester will claim about a leading
player's capabilities vs the reality, which of course is a state of affairs
large Enterprise software companies are all too keen to encourage.

I've never seen a sales consultant answer "no" to a question about capability
in an enterprise software sale scenario, ever, across many RFPs. If your job
explicitly targets you to bring in x millions of dollars of business a year,
you answer "yes, of course" safe in the knowledge that when it does blow up
it's someone else's problem and far away from you by that point.

~~~
kabdib
Duplicity is the name of the game in Enterprisey software. Whether you're a
startup struggling to add the features that a client requested, that sales
said "Of course we can!" to, and that are utterly foreign to your product
("Why the hell did they ask for voice recognition in this vacation hour
tracker?"), or a hugecorp like Oracle with a do-nothing platform that is
essentially re-written on site to customer specifications, the time-honored
practice of "managing customer expectations" is the name of the game in this
market.

Falling behind? Ship half a feature, lie about it or explain how it's going to
be working in the next release (and when that doesn't happen, blame the
customer's requests for churn, or the height of the tide, it doesn't matter).
In danger of actually completing a contract? Trod hard on the bugs, you don't
want that money pump to stop. Larry Ellison needs a new dock for his yacht?
Time to increment the product's version field and let a thousand consultants
bloom. Ka-ching, baby.

My experience with the space, both as a developer of Enterprisey software at
several start-ups and as a customer, was that this corner of the industry is
ethically sick, and I won't have anything to do with it.

------
manofstick
New Zealand in the 90's was trying to have an integrated policing system
designed by IBM which burnt a lot of money (in the order of 100 million) and
was abandoned [0].

I was working with one of the companies that came along to replace one part of
the system with a much smaller component that just handled that individual
piece and vividly remember the meeting with the IBM team. They had no code to
show me, but they handed my a massive folder - probably verging on 10cm thick
of use case diagrams. So many little stick figures staring up at me I was
awestruck - but not in a positive way!

[0] [https://en.wikipedia.org/wiki/INCIS](https://en.wikipedia.org/wiki/INCIS)

~~~
abraae
Yep, INCIS was a cockup of massive proportions.

I remember interviewing a BA (business analyst) who had just come off of the
INCIS project. She said their entity model was enormous, with tens of
thousands of entities.

I couldn't understand how any project could have so many entities, let alone
how anyone would ever hope to code up such a thing.

She said the commandment to the analysts was to model everything so they did,
just like they learnt in data modelling 101.

So (INCIS was a project for the NZ Police), e.g.:

\- A police station has zero or more cells

\- A cell has one or more doors

\- A door has one and only one key

\- A door has many bars

Another data point in INCIS - my sister at the time worked in the police. Her
workplace ran out of pencils (they did a lot of note taking with pencils and
paper). INCIS had so thoroughly depleted the police's budget that there was
nothing left for...say.. computers with email and the like.

------
erpellan
The real achievement here is managing to run up a bill of 1B CAD while still
failing to deliver. I mean, I can think of a few projects that probably cost
several million/yr but 100x? Astonishing.

There's an anecdote floating around about the consultant who heard of a
billion-dollar government IT project who promptly offered to do it for 20
million. When asked how, they replied that they would sit on a beach for 5
years then pick up the phone and say, "We failed". Their contact got back in
touch 5 years later and said, "I wish we'd given it to you".

~~~
tootie
CityTime was budgeted by City of New York for $63M and after spending $700M, a
bunch of people ended up in jail.

[https://www.wnyc.org/story/143601-citytime-cautionary-
tale/](https://www.wnyc.org/story/143601-citytime-cautionary-tale/)

------
kellysutton
This sounds like a good application of Gall's Law:

A complex system that works is invariably found to have evolved from a simple
system that worked. A complex system designed from scratch never works and
cannot be patched up to make it work. You have to start over with a working
simple system.

------
didgeoridoo
The name is just too perfect: [https://www.amazon.com/Phoenix-Project-DevOps-
Helping-Busine...](https://www.amazon.com/Phoenix-Project-DevOps-Helping-
Business-ebook/dp/B00AZRBLHO)

To that point, I’m wondering if all IT initiatives codenamed “Phoenix” are
doomed from the start, given that the implication is “let’s burn down
everything we have that works, and rise anew from the ashes”...

~~~
projectileboy
Hah! I make this joke all the time with friends. I have literally been on
three different failed projects named ‘Phoenix’

~~~
da02
Why did they fail?

~~~
projectileboy
Well, one possibility is that I am the common factor in all of my bad
relationships. But another possibility is that projects tend to get named
'Phoenix' for a reason, and that just as in real life, it's pretty hard to
successfully be reborn from a pile of ashes.

~~~
da02
What are some projects that succeeded and/or impressed you?

------
vidanay
So, what makes a payroll system at this scale so difficult? On the face of it,
there are salary employees and there are hourly employees. How many taxing
bodies does Canada have? Federal, provincial, and municipal? Is that a few
dozen or a few hundred? Is Canadian tax code crazy complicated that doesn't
allow for a rules based system?

I'm honestly curious as to how a "simple" payroll system can go so far off the
rails.

~~~
mkohler
[http://wiki.c2.com/?WhyIsPayrollHard](http://wiki.c2.com/?WhyIsPayrollHard)

~~~
vidanay
That's.....amazing

It almost gives the impression that an air traffic control system is actually
easier (although also failed.)

~~~
fruzz
Both are incredibly complex (I am a developer on an air traffic control
system.) There are thousands of requirements for voice communications alone.

------
nradov
For historical perspective on another (in)famous payroll system project I
encourage developers to read about the Chrysler C3 project. It ultimately
failed, and yet provided the genesis of some agile development best practices
that most of us use today.

[http://wiki.c2.com/?WasChryslerComprehensiveCompensationSucc...](http://wiki.c2.com/?WasChryslerComprehensiveCompensationSuccess)

[https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...](https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System)

These things are hard to get right in any complex enterprise.

------
blantonl
_The software platform itself, a heavily modified version of Oracle’s
PeopleSoft program ..._

What implementation of Peoplesoft _isn 't_ heavily modified?

Adapting business processes to COTS software packages is extremely
challenging. Anyone remember the absolute _bank_ that Peoplesoft consultants
made back in the day when ERP and Business Process Re-engineering was all the
rage?

~~~
kabdib
When I last "dealt with" PeopleSoft, the standard operating procedure was:

1) Pay PeopleSoft/Oracle a metric f--kton of money.

2) A CD arrives in the mail. (This little envelope cost $800,000?)

3) Your PeopleSoft consultants show up, laugh at that CD (tossing it into the
microwave) and pull out their own distros with a zillion customizations that
they proceed to spray over your servers. You have no idea what they are
actually installing.

4) You get out a firehose, fill it with money, and keep showering the
consultants with cash until you are sure they won't succeed in delivering
anything worthwhile, and you fire them.

I was only installing a local test instance of PeopleSoft (to do some
integration work with another product). Somehow the PS consultants in the area
got wind that someone in my company was doing an install, and I started
getting all kinds of cold calls offering customization and optimization, with
some of the most condescending and customer-hostile attitudes I've
encountered. PS had pretty a pretty shitty ecosystem 15+ years ago, and it
doesn't sound improved. I'd hate for anyone nontechnical to be at their mercy.

~~~
mkstowegnv
I was privvy to discussions surrounding a search for a university president.
One of the candidates dropped out when he heard that the university was
replacing a perfectly functional home brewed accounting system with PS - it
had brought his previous university to its knees and he didn't want to ever
deal with it again.

~~~
kabdib
That candidate showed remarkable common sense.

------
patio11
Cross-posting a comment I made from 7 years ago when NYC's payroll project
blew up:
[https://news.ycombinator.com/item?id=2716319](https://news.ycombinator.com/item?id=2716319)
which I last cross-posted to the thread about Phoenix's payroll project
blowing up:
[https://news.ycombinator.com/item?id=15303555](https://news.ycombinator.com/item?id=15303555)

------
dman
Could this be the first time someone gets fired for hiring IBM? (Sorry could
not resist after having heard of that quip so many times).

~~~
nikanj
Probably not. IBM will return a token amount of money so politicians save
face. Someone might take the revolving door either direction.

------
bloomingfractal
and that's why governments should hire software engineers that can be in the
loop and understand not only the technical but the policy side. Hiring an army
of contractors earning 300k/year to deliver "something" in the waterfall model
is a recipe for disaster.

~~~
threeseed
Have you actually tried what you are saying ? Because it's a nice idea just
completely unrealistic.

Software engineers who are not just technical but SMEs as well as having
excellent skills in stakeholder engagement are incredibly rare. And most of
them know how good they are and are contracting at, you guessed it, around
300k/year. Talented engineers are usually quite savvy when it comes to money.

Also for most projects like this it is Waterfall for the overall project
(Requirements, Design, Development, UAT, Production) and Agile for the
Design/Development parts. It's really the only way since the client does
Requirements/UAT/Production and the vendor does Design/Development.

~~~
dragonwriter
> Have you actually tried what you are saying ? Because it's a nice idea just
> completely unrealistic.

I've been in places where it's done. In fact, I've been involved for a long
time in a public sector environment with three key systems, one of which is a
central claims adjudication system, and two of which are accounting and
tracking systems that interface with it, performing similar functions to each
other but for two separate programs that share the adjudication system. Those
three systems, for historical reasons, have three separate models:

The adjudication is state-owned (that is, the code belongs to the state) and
operated, developed almost entirely by a team of contractors with some
involvement by state programming staff and a state development lead and also
tight integration state-staff IT team that does requirements analysis and
acceptance testing and manages most of the interface with state business
users.

One of the accounting/tracking systems is, and has been from the beginning,
developed and maintained almost entirely in-house by state staff (and almost
solely with programming staff.) Because of subsequent organizational
consolidation, the state team working on this system overlaps the one working
on the prior system.

The other is a vendor-maintained MOTS system, with state IT oversight.

Each has it's strengths and weaknesses, but the MOTS system is consistently
the one that is the roadblock to adapting to changing business requirements,
and both before and after consolidation the all-internal one was the one with
the quickest best-case requirementd to delivery speed, and by far the least
expensive for the value delivered.

~~~
da02
Was this at the city, state, or federal level? Or in another country?

~~~
dragonwriter
Large state.

------
giarc
So it doesn't have to be this way.

I work for Alberta Health Services, we have 120,000 employees across the
province. There are likely 5 separate unions plus out of scope staff
(management). This organization was created after the merger of 5 separate
health care organizations all with separate systems etc. They over time rolled
everyone into the same pay system and I didn't hear about a single person
missing their pay. The federal government transition has resulted in tons of
people being not paid at all, being paid too little or being paid too much.

------
lopmotr
"there is reputational risk for IBM in not helping us fix this,”" \- not true.
IBM has a long history of exactly this kind of problem. Canada didn't care
about their existing poor reputation and the next government to hire them
won't either.

------
endorphone
The Canadian government attempted to build a gun registry, originally
predicting it to cost $12M, then $85M, then $1000M, then $2000M, and then it
was scrapped. While it's easy to blame vendors like IBM, I have to imagine
feature-creep and "what about"-isms lead to such monstrosities.

~~~
beamatronic
How many long guns are there in Canada anyway. Could you not use
sheets.google.com? I'm serious.

~~~
walrus01
There are quite a lot, actually, though canada has a greater proportion of
traditional type long guns like bird hunting shotguns, bolt action rifles,
etc. Less people who want to buy a consumer-legal version of a HK416.

------
myrandomcomment
This is sad. IBM used to be the watch word for "It just works!" I started at
IBM many years ago. When I went to my 1st ISP (back in the early 90s) and
things broke I was like "what do you mean it broke? How is this okay?" It was
quite a shock to move from a company where we had systems with 20+ years of
uptime to having reboot the USENET server every 5 days.

------
j45
The comments made about the government about excluding IBM from making future
systems for the government stood out.

The reality is payroll should be executed by payroll specialists. I recently
had to research payroll providers and there are players who service 20-30% of
the Canadian population already.

This is another sign of folks not knowing how to source, design, oversee or
implement software implementations.

~~~
52-6F-62
This is also why Ceridian is still around in spite of all of their faults (or
so I hear from any small/mid-size business practitioners I know).

I take it that's who you're referring to?

~~~
j45
There's actually a few at that size, Ceridian is one, ADP is another...

Most have their pros and cons but ultimately can handle complex union type
payrolls.

Payroll, also is ripe for disruption.

~~~
52-6F-62
>Payroll, also is ripe for disruption.

From everything I hear about working with those companies—new and suitable
replacements would be welcomed wholeheartedly.

That is, if they can maintain the same level of professionalism and ability to
handle situations like you mentioned. Reliability is of immense importance, so
it's not as easy to just hack away at it and sell people an MVP and iterate.

~~~
AndyNemmity
Ripe for disruption, except that when you look at the laundry list of
requirements, the thing that needs disrupting is the need for a billion
different custom requirements, not a different payroll system.

~~~
j45
I thought that too, until I learnt how all the payroll rules are fed into a
mainframe for companies like ADP and Ceridian.

A mainframe stood between a client and their need.

Maybe we're after a better mainframe that doesn't need so much workarounds to
do what has become payroll today.

I'm choosing to be optimistic that if a Stripe can navigate the payments
industry, a payments startup could allow a better mainframe to be built
through a rules engine that allows the degree of flexibility and data
interoperability required in today's world.

------
Analemma_
It's been a _long_ time since I've heard about any successful projects coming
out of IBM's consulting and services businesses. Mostly it's one headline
after another like this- "Billion dollar project scrapped". How much longer
can they keep coasting on the reputation of what was, if we're being honest, a
completely different business that just happened to have the same name?

~~~
AndyNemmity
You don't hear about successful projects. They don't make news stories.

I've done a ton of projects in consulting, not IBM, but you've never heard of
any of them, because they worked, and the customer was happy (although maybe
some delays, or other randomness.)

It's self selecting that news is about massive failures.

------
walrus01
Canadian here. This has become a huge political football. People are blaming
the current Liberal government, when the project was started by the Harper
Conservatives. The full extent of just how fucked up it is has just now become
public knowledge.

It is also a political pork barrel project.

> The project was meant to save costs by firing 1,200 employees handling
> payroll at various departments around the country and replacing them with
> about 500 people in a centralized location using Phoenix to handle most of
> the government’s payroll needs.

What's not written there is that the decommissioned long gun registry was
located in a rural part of New Brunswick. When it was killed by the
conservatives, they had a big political issue on their hands because suddenly
hundreds of federal government clerical workers were out of a job in this
area. So what was the solution? Re-hire the same people and put them on the
Phoenix payroll project. Never mind that not one of them had any experience
with IBM, Oracle, PeopleSoft, payroll, finances, or anything else related.
They wanted butts in chairs in front of keyboards and to keep those people on
the federal workforce payroll.

[https://www.google.com/search?num=100&client=ubuntu&hs=ssW&c...](https://www.google.com/search?num=100&client=ubuntu&hs=ssW&channel=fs&ei=6leYWpieE4PgjwOUsoGIAw&q=long+gun+registry+new+brunswick+miramichi&oq=long+gun+registry+new+brunswick+miramichi&gs_l=psy-
ab.3...6322.8176.0.8318.10.10.0.0.0.0.143.918.5j4.9.0....0...1.1.64.psy-
ab..1.8.778...33i160k1j33i21k1.0.czpxz7HDuMU)

quick edit, history lesson:

early 1990s, Chretien Liberals: we need a gun registry

liberal party: okay we passed a gun registry law

liberal party: we need a pork barrel project to make these people in small
town new brunswick happy, let's hire 500 people in miramichi

liberal party: this gun registry has now cost 400% of what was originally
budgeted

liberal party, paul martin government loses an election to stephen harper and
the conservative party

harper conservatives: we promise to scrap the long gun registry

harper conservatives: we've removed the long gun registry but now all these
people are out of work and pissed off

harper conservatives: we need to overhaul our old payroll system that runs on
a bunch of old minicomputers, let's hire ibm

harper conservatives: let's re-hire those people we fired and put the payroll
center in small town new brunswick!

harper conservatives lose the 2015 elections

trudeau liberals: wtf is going on with this contract the previous government
signed. wtf is going on with this clusterfuck of oracle, ibm, peoplesoft and
porkbarrel politics of employing people who are not payroll specialists.

trudeau liberals, later on: seriously this is broken beyond repair

~~~
Quinner
The way this article was written was really weasely.

> The Phoenix project was originally chosen by Prime Minister Justin Trudeau’s
> Conservative predecessors

This is assigning blame to Trudeau in the weirdest way.

~~~
hueving
It sounds like it's blaming his predecessors?

~~~
bsiemon
Still mentions his full name and title front and center. I think that is what
gp was referencing.

------
fnordsensei
It's possible to infer from this that CA ordered an output from IBM, rather
than an outcome.

IBM can claim that they have indeed delivered the _thing_ that was ordered,
while CA remain disappointed that the _effect_ they had in mind is missing.

In Canada's defense though, it's sometimes hard to find someone who will even
take on contracts of this magnitude. Getting good terms, even more so.

~~~
jandrese
These large government software projects end up costing way more than they
should pretty much every time. The process is just broken. There's way too
much design by committee and tying the hands of the people who are doing the
actual work, preventing them from implementing the correct solution and
instead making them implement the one the disconnected-from-reality
specification writers decided on.

Then they have to go back an iterate the correct solution, which involves more
committee meetings at huge expense and rewriting documents and disseminating
them to everybody...

~~~
threeseed
This is completely wrong.

The CA government engaged the vendor, IBM, after they had decided on a list of
requirements from what they wanted from a solution. In your argument you say
that the hands of the people doing the actual work were tied. Well that's the
vendor, and so no, their hands weren't tied.

And not sure if you've ever worked at a project like this but the committee
meetings aren't expensive. They are basically free. And there often should be
far more of them. The real cost comes from major changes in the design too
late in the project plan.

~~~
jandrese
Committee meeting take the time of multiple people. They are absolutely not
free. People's time is by far the biggest expense in a development project
like this.

In the long run they can save money (getting everybody on the same page to
avoid conflicts in the future), but you're paying up front for that. In
practice the value of the meetings decreases with frequency (in the reducto ad
absurdum case 100% of your time is spent in committee meetings and nothing
gets done).

Only engaging with the vendor after the requirements were drawn up is what I'm
talking about. They contract with IBM and hand over a 2,000 page document
describing their requirements, then discover that it's impossible to implement
and end up spending many years and millions of dollars on trying to fix it.

------
cm2187
Hard to have an opinion without knowing the details but from my little
experience I feel like the reason why these big project fail is not so much
because they are too complex, I am sure a good (not excellent) developper
could come up with an algorithm and data structure generic enough to
accomodate all corner cases. But because these big projects start from a
preexisting solution (PeopleSoft in this case), that wasn’t designed for this
particular complexity, so it requires orders more energy to go around its
limitations, and the people staffed by the likes of IBM on these projects are
usually project managers with little to no programming skills and little
domain knowledge. So you end up with a disaster for things that aren’t that
complicated (custom shifts for coast guards and prison guards doesn’t look
like a hard computer science problem).

------
tboyd47
> The project was meant to save costs by firing 1,200 employees handling
> payroll at various departments around the country and replacing them with
> about 500 people in a centralized location using Phoenix to handle most of
> the government’s payroll needs.

> The government will spend C$431 million to keep the program running in the
> near term, on top of C$460 million already spent to put Phoenix in place and
> fix the problems it generated.

So for each of the individual salaries Phoenix was supposed to eliminate, they
spent C$600k ($500k USD) in up-front costs, and it will be another C$600k to
clean up the mess. Turns out, software takes a lot of time and money to build.
Who knew? I guess IBM didn't. Or maybe they didn't care?

Maybe there should be some kind of advisory board system for large software
projects, made up of disinterested-yet-well-informed techies, similar to what
exists for corporations.

~~~
dade_
There is some missing math. They likely have 30 or 40 payroll systems all up
for renewal/replacement. The old systems are probably loaded with features and
forgotten requirements that would have been discovered during deployment.

~~~
pixl97
Govt.office "We need to open these X year old databases/files for tax purposes
once a year"

Programmer.ibm "What the hell format is this?!"

------
jorblumesea
The whole big 4 IT consulting industry needs to die a swift death. It's
basically institutionalized corruption. Overbill the client, under deliver
with shoddy products made overseas, get that vendor lock in and watch the
money flow in. The cost savings are largely an illusion sold by false promises
of a better world.

~~~
Demoneeri
You are mistaken, I work for one of these. I have 12 years experience and I
have only one failure to deliver and it was stopped in the design phase
because it was too complex and the client wasn't ready.

~~~
jorblumesea
I worked for one too and I can't say the same thing. Entire projects were
poorly thought out, designed and improperly executed. The people working for
the company were incompetent and their "hiring bar" was if you had a pulse.
And this was a big well known firm.

~~~
AndyNemmity
You're both right, I've seen both sides.

The reality is, in all of them there are teams of amazing, insanely talented
people, and teams of complete train wrecks.

It's just life. In any large company, there's a massive variance in ability.

------
makecheck
Sometimes I wonder how these projects are started; is it really that they go
straight to “here’s a binding contract for millions”, without having them do
any smaller projects in between?

If it were me, at the very _least_ I would start with something small and say:
“deliver this, let me see how it works, give me every last line of code for
further inspection, then we’ll talk more”. I mean, have them build a mini-site
or something and make sure that they’re competent before unleashing them on
_the entire payroll system_ , right? And you don’t need some grand familiarity
with the software industry to plan that way, it just makes sense to start
smaller and make sure you know who you’re really dealing with (and also
spending less money if it fails).

------
guyzero
I hope to someday read a comprehensive post-mortem but I doubt there's anyone
involved who can/would write one without a lot of bias against the other
parties involved. This will go down in an annals of massive software project
failures.

~~~
eyrarric
Actually, Australia went trough the same payroll procurement fiasco with IBM.
You can read the report
[http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf...](http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf_file/0014/207203/Queensland-
Health-Payroll-System-Commission-of-Inquiry-Report-31-July-2013.pdf)

------
redleggedfrog
IBM?! Oracle?! And you got away with only losing $780 million! Consider
yourself lucky.

------
dalbasal
We really need to rethink the metasystems around these kinds of projects. A
$1bn government payroll project is like the canonical "project that is
guaranteed to suck" but realistically this extends pretty widely.

These custom software builds, built around specific corporations, with all the
sales and account management and expectation planning and making sure that
*when" this fails, it is the other party's fault....

We need to find better ways of doing this stuff.

------
toomanybeersies
To add to the list of failed (at least initially) payroll systems, we have New
Zealand's Ministry of Education and their Novopay system:
[https://en.wikipedia.org/wiki/Novopay](https://en.wikipedia.org/wiki/Novopay)

It cost $180mm and was borderline non-functional, and then cost another $40mm
to fix to its now functional state.

The same company that did Novopay also did the payroll for NZ Post and had
similar issues.

------
verdverm
I have payroll issues with IBM still. They overpaid me when I left, then they
got my end date incorrect and wanted more back than they deserve. This has
been ongoing for 7 months, they can't seem to figure it out... just check the
email logs, if they had any... They sent it to collections some time ago, now
trying to get them to negotiate a payoff rather than dealing with IBM Employee
"services"

~~~
djrogers
Whoa, doesn’t your state have an agency that handles that stuff? I was
terminated once by a company that claimed they went bankrupt, so I filled out
one form with the labor commission (had to in order to claim unemployment
benefits - I was 22 and had no savings), and they got ALL up in my former
employers business and straightened it out quickly with no extra effort in my
part.

------
tw1010
This reminds me of the part in the Robert Moses biography when he was made
responsible over implementing a more meritocratic employee review system for
the government. Just like this story, that one also ended in flames, though
Moses made it out all right in the end. I wonder who the Moses character in
this story was, and if we'll read about it in a biography in a few decades.

------
ttt111222333
> Canada to scrap IBM payroll plan gone awry costing $1B. The Phoenix project
> was originally chosen by Prime Minister Justin Trudeau’s Conservative
> predecessors 10 years ago to centralize the government payroll.

Is it me or is this wording terrible?

Trudeau is not a Conservative. It makes it sound like it is Trudeau's party
that caused the problem.

~~~
mattnewton
Yeah why is his name there at all?

~~~
bcholmes
And why isn't "The Harper Government"(TM) not named?

~~~
loufe
I'm not a fan of the convservatives but it's still unfair to place the blame
of a bad choice of client on a political party. A lot of non-partisans were
involved in getting the project running.

~~~
tempestn
Sure, but it would make more sense to refer to them by name than as "Justin
Trudeau's Conservative predecessors" or whatever.

------
PayrollIssues
So: for anybody who is curious what's actually going on here:

1\. Problems are far bigger than the specific payroll software system. There
were tens of thousands of outstanding cases in 2015 before the new system went
live. There were tens of thousands of outstanding cases in 2012 when the
project started. And there were tens of thousands of outstanding cases every
year before that on the "old great system". (why? read on)

2\. IBM didn't create a new system from scratch - Govt chose PeopleSoft HCM,
an industry-leading system which reliably pays people day in and day out at
thousands of customers. Base solution is not the core issue.

3\. When you buy PeopleSoft (or SAP, or JDE, or whatever), ideally you make
your business fit the system's best practices, customize as little as
possible. No client fully does it, but govt doesn't even try. IBM, together
with govt employees, spent five years trying to hammer several tens of
thousands of complex time and labor rules from more than hundred labour
agreements into the system.

3\. Implementing this system was half of the initiative, other half was
reducing workforce for compensation advisers and moving them all to NB, hiring
new grads as replacements. Who were surprisingly not exactly intimately
familiar with these hundred labor agreements and tens of thousands of time and
labour rules. We're talking one third of the support force at one tenth of
experience.

4\. So, the billion wasn't spent on "the system". It was mostly spent
scrabling afterwards to re-hire old CA's as expensive consultants and trying
to catch up the damage.

5\. Because guess what, garbage in, garbage out. If your manager doesn't
submit or approve your acting gig or leave of absence or maternity leave for
several weeks, or months, or more, system won't magically know about it. Not
the old one, not the new one, not some magical future one.

If your HR person doesn't create you as a new employee, you won't get paid.

If you don't submit your overtime slips on time, but instead collect them for
10 months so you can have "Xmas bonus", that also MAY just may create an
issue. Ditto with retirement applications. Secondments to other departments.
And if through this all employee groups perform denial of service attacks on
the system because this is all a political game, that may also be a problem.

There is a pervasive cultural issue that boggles mind as to why nobody does
their HR things on time or correctly there. The system just provides useful
target of the moment, but issues were there before and without cultural change
they'll be there in the future.

There are real problems and there are real people suffering because of them,
and they have for decades, and yes it's gotten worse. Grandstanding over some
magical new systems won't fix it though.

This is a burner account, I had the dubious privilege of working on this few
years back and am aware of some of the horror stories - from the present and
the past. I'll stay the dear heck away from Federal Govt / Public Sector
project in the future if I can.

~~~
d_runs_far
As a manager who is on the receiving end of GoC payroll, you make some
excellent points. The key ones are: 3 - reduction of corporate knowledge. My
pre-phoenix comp advisor could make shit happen. Now, I can't even get a
return phone call.

Likewise, pt 5 (and 3). Example; pre phoenix, our summer students would always
have a pay glitch end of July - why, comp advisor on holidays, didn't properly
brief their replacement, stuff fell through the cracks.

I think the scale of the problem and suffering is much greater with this
implementation than it was under the disparate departmental systems.

Not to be political, but, really that plays into this while conversations is
that what I don't think can be ignored in all of this though is that the
Harper mentality of consolidate to save money hasn't worked in any regard;
just look at Shared Services - try getting timely procurement, and oh, BTW, no
spares allowed to be kept... complete clusterf... as well. That's a whole
other mess that affects people in a different way - oh, your phone line is
down; tech will be there in 2-3 months.

------
hindsightbias
"ADP tech chief Stuart Sackman says his company is under constant attack
globally and is hoping to encrypt more of its data using the new technology. A
sixty year customer of IBM, ADP currently runs its largest domestic payroll
engine on the systems, calculating 20 million paychecks per pay period, while
also running its money movement, tax filing systems and cloud-based software
suite on versions of the Z-series mainframes."

[https://www.forbes.com/sites/alexkonrad/2017/07/17/ibm-
seeks...](https://www.forbes.com/sites/alexkonrad/2017/07/17/ibm-seeks-boost-
with-hacker-proof-mainframes/#7304138ca9e8)

~~~
Toast_25
As someone working in ifosec, i _loathe_ how vague that article is about what
exactly is going on in the chips.

------
CivBase
Does anyone know what the real problem was here?

> Some people were paid too much, others not at all. Those issues snowballed
> with the deluge of requests to fix incorrect paychecks.

> IBM said it’s “fulfilling its obligations on the Phoenix contract, and the
> software is functioning as intended.”

Okay, so the cost is coming from incorrect paycheck values, but the software
is working correctly?

Does that mean it's human error? Can these issues be traced back to one or a
few major instances of human error? Is this a widespread issue caused by a
confusing or misleading UX design?

The article is quick to talk about the projects costs and failures as well as
the politicians involved, but I can't learn anything or blame anyone until I
understand what went wrong.

~~~
tempestn
From the anecdotes I've heard, it sounds like the support staff is just wildly
insufficient to handle all the issues that crop up with a major change like
this. Initially the government switched over a trial group of (iirc) something
like 40k employees. There were a bunch of problems with people not being paid
or being paid the wrong amounts, but for some incomprehensible reason (perhaps
they wanted to ditch the old system to focus on fixing the new one or
something crazy like that?) they switched over everyone before fixing those.

Now, I personally know people who've been getting incorrect pay for over a
year, who submit a complaint to their HR, who pass it up the chain, where...
nothing seems to happen. Presumably because the 500 people or whatever it is
tasked to deal with this are literally dealing with tens of thousands of
issues, many of which aren't trivial changes (and so block the many that
actually _are_ trivial changes, but aren't even being seen).

Basically it was released before it was ready, initial signs of issues were
ignored, and now the human side of the system is overwhelmed. It's kind of
like when a web server gets overwhelmed with traffic, and it isn't set to
reject at a reasonable threshold, so things start getting queued and timing
out, people start refreshing pages, and it just snowballs until the whole
thing goes down. Which apparently is what has now happened.

~~~
grafporno
> a trial group of (iirc) something like 40k employees

Oh boy.

~~~
tempestn
Don't quote me on that number; it's from memory. Point is, it was a fraction
of the total, and then they released it to everyone without solving the
problems in the trial group.

------
kitd
Totally OT but

    
    
        The Phoenix project was originally chosen by Prime Minister Justin Trudeau’s Conservative predecessors 10 years ago 
    

seems an odd way to refer to Stephen Harper's government.

------
dgregd
The only solution to tax / accounting law and software problems is to create a
reference accounting system and set it as a applicable law. In other words, to
describe the law in unambiguous language.

Do we really have to write a law in the same way as Romans did? Math as a
science made progress after it invented a new precise language for itself.

I'm not talking about the whole law. Only about that according to which the
software is built. Since no one calculates taxes today by hand, it is worth to
write the law so that it can be easily applied.

------
harry8
Ibm global services, Accenture, computer associates, Oracle. See a company
sign a big contact with them or their ilk, short that co. Their whole raison
d'etre is to rip off old assholes in the c suite who can't find the on switch
of the company or government money they're meant to be protecting.

Hiring these consulting firms should get you fired every single time. It's a
wonderful indicator of total incompetence. Notice how they're all deep into
Washington? Yeah. Makes me sick these companies exist.

~~~
mrbill
Two common factors in a lot of these stories are "IBM GS" and "Oracle"...

------
Murkin
Would you sign to build a project who's MVP is 260,000 salary calculations
with rules poorly documented and spread other hundreds of agencies?

Maybe if they started with a subset and slowly built up..

~~~
cjalmeida
This problem was likely better served by many smaller systems that better fit
the specific needs.

Government is way too complex to try to cram all rules into a single monolith
like Oracle's PeopleSoft.

~~~
rpeden
If what I've read is accurate, the various departments were already served by
smaller systems that fit their specific needs.

The intention with Phoenix was to save money by consolidating them all into
one. Oops.

At least part of the problem is that projects like this don't end up being
awarded to those who are most capable of completing them successfully.
Instead, they end up being awarded to companies and consultancies who are
experts at shepherding proposals through the RFP process.

------
dagw
Anybody have any insight on what it feels like working as consultant on such a
huge failure? I (or my department, but honestly mostly me) once failed, as a
consultant, to successfully deliver on a $50k contract and I felt like
absolute shit for weeks and had trouble sleeping. I'd hate to think how it
would feel to fail on this scale. Or is it a situation where the whole thing
is so large and diffuse that no one can honestly be said to be responsible.

------
bitmapbrother
I'm surprised there weren't any financial penalties for the IBM practice of
over promising and delivering a broken system that generates more problems
than it solves. They underbid and embellish proposals with BS to get the
contract, staff if with overseas workers to get the maximum return and then
hope for the best. This is how IBM does business and it's unfortunate that
they continually get away with it by profiting on their incompetence.

------
moltar
The Big Rewrite. What can go wrong?

------
ismail
I am guessing one of the reasons is probably the software selected... oracle

~~~
dsfyu404ed
Oracle/PeopleSoft can do payroll for a defense contractor with tens of
thousands of employees across multiple nations.

I think proper configuration is what's lacking in this case.

~~~
debacle
Oracle costs a lot. It infects your entire stack, from hardware all the way up
to personnel, and drastically inflates the cost from end to end.

With no real utility advantage, either.

~~~
threeseed
Everything infects your entire stack. If I hire Java, Ruby, PHP etc people
then everything from hardware up to personnel is going to be selected to
optimise the delivery and execution of software. Kind of common sense that you
would do this.

I don't disagree that Oracle costs a lot and that they inflate margins pretty
poorly but to say they have no real utility advantage is a bit ridiculous.
Most of the world's enterprises are running core business systems off Oracle
and if they move to the cloud it's usually to Oracle instances in the cloud
not any other system.

------
maerF0x0
I wish governments would just get on with it and open source their
functionality for those of us who want to "serve our country"

~~~
dragonwriter
Passively wishing does not tend to influence government policy.

~~~
maerF0x0
Don't mistake my HN comment as being my _only_ action.

------
nlolks
Have you heard of Open Source? Government of France has. Enjoy the article.
[https://www.enterprisedb.com/blog/france-takes-lead-new-
open...](https://www.enterprisedb.com/blog/france-takes-lead-new-open-source-
front)

------
rmrfrmrf
I've never seen a PeopleSoft implementation go well. It's always underbaked
and overpriced.

------
tandr
I am genuine curious - Are there any recent examples of big, long but
_successful_ projects that involves these 2 companies? (IBM and Oracle are
mentioned in the article)

------
ll931110
There seems to be a gap to fill for companies/organizations building software
for government. Wondering which reputable companies are filling this gap?

------
S_A_P
Former ibm employees are even toxic. The cmo of openlink is a former ibm cloud
marketer that has effectively run out the entire marketing staff

------
mathgladiator
I wrote my own payroll system for my house keeper, and it is surprising hard
to get taxes right... I am not surprised.

