
IBM contract cost for failure-plagued Phoenix payroll system jumped to $185M - found_reading
http://www.cbc.ca/news/canada/ottawa/phoenix-ibm-contract-union-pay-government-1.4295827
======
bhouston
Like all IBM projects now day the bulk of the work was done in India for
likely less than $5 CAD/hr, while they billed out $100 CAD/hr or even more.

The actual numbers may be slightly different, but still these are close to
reality.

What a great profit margin for a failed project.

I wouldn't be surprised the code is like a lot I've seen from Indian-based
sweat shops (btw there are amazing Indian coders, but you generally won't find
many of them in these software houses producing the run of the mill code) -
cut and paste code that hard codes every option and combination. And who cares
because IBM makes money every time someone asks for a change.

IBM is the winner here, and everyone else is a loser.

~~~
throwaway_45
Why doesn't Canada put a rule saying all work must be done by Canadians. Even
if its cheaper keeping the money in country seems better than sending it to
India.

~~~
microcolonel
> _Why doesn 't Canada put a rule saying all work must be done by Canadians._

Because the government doesn't want to be seen as xenophobic. One of the major
talking points of the prime minister and his government is "diversity is our
strength". They also may want (depending on the department) to be seen as
distancing themselves from expenses, not necessarily engaging in cost savings
per se, but in cost-saving-style behaviour.

> _Even if its cheaper keeping the money in country seems better than sending
> it to India._

Having worked with IBM-sourced contractors working in the U.S. and Canada, I
can say that at least the ones I worked with were pretty useless regardless of
where they come from.

The problem is that the government is bidding by familiarity rather than
competence, and isn't willing to hire anyone competent enough to build a
genuine team. When they tried their _Shared Services Canada_ thing, it was a
disaster.

The problem with government in Canada is apathy. Nobody, and I mean nobody,
cares at all about Canada as an ideal or concept; nobody really cares all that
much about their fellow Canadians.

~~~
Roritharr
> The problem with government in Canada is apathy. Nobody, and I mean nobody,
> cares at all about Canada as an ideal or concept; nobody really cares all
> that much about their fellow Canadians.

It feels the same in Germany, if i had to guess i'd say most western countries
are going through something like this at the moment.

~~~
microcolonel
> _It feels the same in Germany, if i had to guess i 'd say most western
> countries are going through something like this at the moment._

Yeah, that's what I think I'm seeing too. When I talk with my fellow
Canadians, especially those who grew up here, they say things like "we don't
have a culture"; I get a similar vibe from visiting Germans, and even (to a
lesser extent) some less-patriotic Americans.

It's like the west simultaneously forgot that their greatest cultural artifact
is their societies. Western society is _very, very different_ from how people
lived before, and I'd say for the better, but there seems to be a taboo on
taking pride in it.

I tend to get along well with the parents of second gen. Canadians. The
parents seem to know why they came here more than native-born Canadians know
they want to stay.

------
mrweasel
>IBM Canada was the only company to bid on the government's pay modernization
project

If only ONE company bid on your project, then you honestly need to rethink it.
Does the government really believe that IBM was the only company interested?

Any project that only receive one bid should be cancelled and rethought. If
your payroll regulation is so complex that only one company wants to bid, then
maybe you should address that issue first.

Governments are in many cases completely clueless about software and believe
it's some magical sauce that once applied correctly will make up for poorly
thought out and overly complex regulations introduced by the politicians. They
fail to understand that the developers are required to understand the rules
and procedures before they're able to implement them correctly.

~~~
emerongi
It's not unusual for governments to create such project requirements that only
one or two companies can bid on it. The "winning" company is known long
before.

Happens in all sectors.

~~~
thaumasiotes
I believe the practice is known as "single-sourcing".

------
bpyne
"But when it comes to Phoenix, Aylward isn't focusing his blame on IBM. Like
the procurement experts, he faults the scope of the initial contract for the
resulting failure."

Twenty-six years of developing software, I haven't worked on a project yet
that was any better than 60% specified. This inability of people to specify,
in full and in detail, what they want is the reason for the rise of Agile.

A colleague of mine sagely said, "We need to show them something quickly just
so they can tell us what they don't want."

~~~
danmaz74
Golden words.

------
myth_drannon
Another "failed" project for Canadian government, this time by Adobe.
[http://www.cbc.ca/news/politics/federal-government-to-
downsi...](http://www.cbc.ca/news/politics/federal-government-to-downsize-
failing-canada-ca-project-1.4202563)

Since the projects were done by consultancies you can't call it a failure from
the consultancy point of view. They managed to milk the government for
multiples of the estimated budged and this is a great success for them. That's
just how it works - underbid, lock in and then massage the client for x10 of
the original budget.

~~~
petra
Is there a better way for governments to manage such processes ?

~~~
dastbe
One of the major problems (at least in the US) is that the process is
heavyweight and specifically tuned towards "lowest conforming bid". You end up
getting subpar work and long delays, but the government is absolutely
unwilling to restart the process in almost all cases because the cost is just
that large (and we picked this one because it was going to be the lowest
cost!). In some ways, the government has very little control over the outcome.

We've seen through things like 18f and usds that the government can adequately
and effectively manage software projects itself and produce amazing results.
Giving the US government more agency to self-fund ongoing development and
procure solutions that are technically superior as opposed to "lowest
conforming bid" is what is really needed IMO.

~~~
Frondo
The "lowest bid" thing is actually now starting to change. Government buyers
are starting to get trained--and there is an accompanying culture shift--on
the idea of best value, which is a combination of price, timeliness, ability
to deliver, and whatever other factors actually contribute to money not being
wasted.

It's stupid that for so long it's been coded into law that you need to spend
as little as possible, because that's part of what got us here; the buyers in
a lot of cases _can 't_ go with a better, more expensive option.

Hopefully the "best value" culture shift in government buying will go along
with legal changes to let them act on that.

~~~
otakucode
Still the problem is that when Lockheed Martin rolls in with a $350million
bid, Boeing comes in with a $365 million bid, and Joe Schmoe's Web Shop shows
up with a $450k bid, they dismiss Joe Schmoe as trivially worthless despite
the fact he's got low overhead, had built systems exactly like what they need,
and could do it quickly. LM and Boeing, on the other hand, are bumbling fools,
giant towering piles of middle managers and upper managers and the worst
examples of bureaucracy you'll ever face. The kind of people who if faced with
a situation where the company has already been paid but the work location is
going to get shut down for a day they will try to trick workers into taking
their own vacation time to cover it, simply conveniently not mentioning that
the time will be paid anyway. Sure you could just be decent, but if there's a
chance to gouge, GOUGE AWAY. But they look 'respectable'. I've seen a billion-
dollar contract where they found an off-the-shelf product that did exactly
what they wanted... so they bought it, gutted it, and rebuilt it more terrible
than ever in order to justify the expense. I've been on one side saying 'you
guys will mess it up, just let us develop the new version' and been told 'you
can do it... but you still have to pay US for writing it. Because that's part
of the contract we won and we'll sue you to death otherwise.'

~~~
sbov
Part of the problem is random bureaucrats have no way of knowing if Joe
Schmoe's Web Shop can actually do the job. Some of it has to also be covering
your ass - if there are serious flaws in the system, your ass is covered if
you spent $350 million - not so much if you spent $450k.

I have no idea how the process goes, but it seems like some sort of outside
consultation on picking appropriate bids might help. Maybe even something like
the congressional budget office, but for software systems. That way expertise
can be concentrated and they can focus on their core competency, rather than
requiring every agency to develop people who can properly analyze bids for
software systems.

------
otakucode
Government contracts create perverse incentives. I've worked in the space for
the past couple decades, and it's revolting. The government takes bids for a
project - a project written in very specific ways to guarantee that only 1 or
2 companies on the planet are capable of bidding on it, usually. I wouldn't be
surprised if a necessary condition of the IBM contract was intimate long-term
experience with the exact system they were replacing, which was likely built
and maintained by IBM. Then once the contract is going on, if the company is
incompetent and can't meet their schedule - they get more money. At NO point
will the government EVER dare to hold a companies feet to the fire and tell
them 'you must finish this project and absorb the entire cost yourself' which
is exactly what NEEDS to be done.

Instead, companies are incentivized to hire the dirt cheapest labor they can
because getting the job done quicker is actively harmful to their business.
Then at the end of it all.. there's usually a maintenance contract to be
awarded to keep the system upright and operating. The company doing the dev
work is going to bid on that and have an apparent advantage (only apparent
though... if they don't get it the actual technical people who matter will
just go with whoever gets the contract, the contracting company will never
actually relocate anyone) and the worse the system looks, the more bug-ridden
and unlikely to succeed it is, the larger the maintenance contract will be.
Even more incentive to build a ball of shit. Literally at every single turn,
doing whatever makes the software worse is what is most profitable in that
space. And eventually some news articles like this will get published, people
will complain, and someone on the govenrment side of the fence will decide
they want to make bringing the system in as a badge of honor in their career -
so they will slash all the mandatory requirements the system doesn't yet meet
and sign an agreement that whatever half-functional garbage they managed to
get was a full completion of the contract. And then the executives of the
contracting company go out and celebrate at an expensive bar.

I once worked on the maintenance side of a system that had just been
delivered. The original contract that had been granted was a single contract
to develop 3 systems. 3 contracts later, 1 system was delivered. And that
system required 9 months of full-time development work to be ready to be used
operationally. You know what that's called? An unfettered stellar monumental
magnificent success... as far as the contracting company is concerned.

------
muro
Similar thing happened in Queensland,Australia. IBM is barred from bidding on
contracts (actually, the government forbade anyone on awarding contracts to
IBM). IBM sued and won, but the government still will ignore all IBM bids. I
only wish they would be able to do more than just ignore IBM, a few stupid
bureaucrats signed a bad contract cost Australia too much.

Then again, working in the same way as every other bureaucracy.

------
raesene6
The public sector in a number of countries seems addicted to these large-scale
high risk forms of procurement.

Instead of running a larger number of smaller projects, which would be harder
for them to adminster, there's a temptation to lump it all into one mega-
project and shift the responsibility to a consulting company.

The problem with that approach is, of course, that its almost impossible to
accurately specify all the requirements of the mega-project up front, so there
are lots of amendments, and as any consultant knows amendements are where you
make your money as by the time they come along the client is committed to the
project, so you have the advantage in negotiations.

Unfortunately the litany of failed projects that follow this model, doesn't
seem to deter public sector organisations from following them.

I don't doubt similar things happen in the private sector, but failures in the
public sector tend to be more visible.

~~~
leejoramo
Indeed many private sector failures are never known to the public.

One small city newspaper burned well over $4 million on business software this
was never know to the public. The software caused the repeated delays in
delivery, which was explained as a mechanical problem with the press.

At the time the same paper, ran a highly critical story of the county
government for a $100,000 software project over run.

~~~
otakucode
Perhaps attaching a necessity of transparency to the contracts would help?
Like, 'to bid you have to provide a publicly accessible full accounting of the
status and progress of the project during its performance'?

------
kfk
The mess is probably on both sides. The issue is that management doesn't care
about IT until everything breaks apart. This means IT is seen as cost and not
worth the trouble to do well and keep in house. Then of course everything
starts falling apart and it's easier to put the fault on the vendors. Of
course IBM has its faults, but I'd guess they are more on the overselling side
(we can do everything and cheaper and better!) than on the pure technical
side.

~~~
otakucode
The contracting company who provides the IT people likewise views IT as a cost
and not worth the trouble to do well. Only small companies give a damn about
technical things. The contracting companies view their value proposition as
mountains of documents, tons of forms, processes, ISO9001 certifications, etc.
The people who write software or work with hardware? All they do is type.
Their work isn't important in any of this.

------
bitmapbrother
IT horror stories involving IBM don't really surprise me anymore. I'm more
surprised by the companies and governments they continue to game with their
low ball bid tactics.

I once worked with a large team from IBM that was staffed with Indians that
did the work both onsite and offshore in India and it was without a doubt the
worst code quality I've ever seen in my career. There were classes with
10,000+ LOC, methods with over 1000 LOC and dead unreachable code that wasn't
even commented out. Speaking of comments, if it wasn't automatically generated
by the IDE there would be none. But the most horrific example of their lack of
attention to detail and code quality was the absolute absence of any unit
tests in this massive project with millions of LOC. The only unit tests ever
written were by me and those unit tests were subsequently disabled/commented
out when some other developer decided it would be easier to disable the unit
tests rather than to modify them to adhere to the changes they were making.
It's a shame more clients don't hire independent code auditing companies to
inspect the code their paying millions of dollars for.

~~~
adekok
I've seen large companies do the same.

Testing? Pfft.. that's for losers.

Design specs? Same answer.

Apparently if you spend enough on sales, you _can_ sell utter shit to
unsuspecting customers. Then, you beat up the engineering team to fix all of
the bugs that the customers report.

------
scandox
It seems to me there is a dollar amount above which any software project must
surely fail. In other words if it costs more than X, then the complexity
implied is too high and the likelihood of success is too low.

~~~
mathattack
Definitely an inverse relationship here. Hard to tell who is to blame here:
the government or IBM. Neither has a stellar track record.

------
eljimmy
I've worked on federal contracts for Canada for a few years in the past. It
really doesn't surprise me that this happened.

I saw failures on every level. The client (government), the PMs, the
developers. It was mind-blowing how much incompetency was infested into the
entire system.

I think the problem with the public sector is that some of them genuinely
don't give a shit about their work and there's really zero repercussions to
them for not doing their job well. They won't get fired like someone would in
the private sector. With that said, I've no doubt some of them do work hard
and are comparable to their private sector counterparts.

But until workers in the government can truly be held accountable, these types
of IT disasters will continue to happen. Heads need to start rolling.

------
S_A_P
Ive been on another IBM run project that started off very similarly for a
large energy company. A few thoughts about this.

1) IBM pulled the same types of stunts on this project and also failed
spectacularly. Its not in the news though, as private companies are reluctant
to admit failure.

2) IBM and other big consulting companies pour large amounts of revenue and
time into relationship management and networking. This keeps the "nobody ever
got fired for hiring consulting shop x" fallacy alive and kicking.

3) There is never a large implementation that is "out of the box and turn key"
If a consulting shop tells you this is possible without great sacrifice and
compromise to business process, you should run the other direction.

4) IBM and other big consulting shops regularly oversell their specialty
domain knowledge, and often end up farming it out to independent vendor/domain
experts after they realize they're in over their head.

5) While I think IBM is a shell of its former self, their behavior here is big
consulting industry standard.

6) Most private companies know that a project will largely come in way over
budget, and often don't care as long as they can afford it. The overages are
usually written off and as long as the operational risk is shifted away from
the business then people are usually happy.

As an independent consultant, Im obviously biased, but my opinion comes from
experience working along side with Deloitte, IBM, PWC, and some other big
consulting shops. I am usually one of a few niche independent contractors that
are brought in to handle specialty items.

So why does this keep happening? Most enterprises/governments do not want to
place their faith in boutique consulting shops or independent consultants as
this seems too risky even if they would likely be able to implement for half
the money in half the time. The arbitrage for big consulting shops comes from
this. Most big consulting companies are going to put 1 or 2 senior and
expensive(to their own payroll) people on the project and leave the grunt work
to cheaper junior onsite or offshore resources. That work is still billed at
150+ per hour and the senior folks 300+ per hour.

Payroll isnt "hard" per se, its just hard to model into a single comprehensive
process. Complicating this is the fact that laws and taxes vary by employee
type and location. Internal business rules vary everywhere, and even though
best practices exist, there is always differences from one place to another.
Finally, people get really pissed when their money aint right...

Ive noticed, anecdotally, that IBM in particular has had issues with big
projects and variances. I chalk that up to being the relative newbie in the
big consulting world, and they're likely underbidding to get a foothold.

------
throwaway5050
Undoubtedly, IBM is partially at fault. However, I strongly believe that a
vast majority of the blame rests on the Canadian government's managers. My
thinking comes from my experience working with government.

\---

I work for a contractor that does the software development work for a large
government entity (albeit smaller than the entire federal government of
Canada). And while I have some incredible horror stories to tell, the
following is particularly bad:

One project was particularly bad. During the initial requirements discovery
phase, it appeared like a simple CRUD app. So, my firm proposed an
architecture that suited the requirements we found. However, when the
government employees in charge of managing the contractors looked at it, they
tacked on a few requirements and changed several requirements/fundamental
assumptions. So my firm adjusted the proposal and resubmitted. The cycle
repeated several times. The government managers really wanted this particular
project out as quickly as possible, so my firm had started building some of
the basic parts of the application (i.e. creating the database tables,
importing the models/relationships into the code base, designing the UI, etc.)
before the architecture was finalized.

After a month with 5 engineers working on the project full-time, the managers
decided to completely change the requirements altogether. So, we had to throw
away everything we had made to date. That month of thrown away work costed the
government $100k+ and they have literally nothing to show for it.

~~~
HeyLaughingBoy
Repeat after me, class, "not properly managing Requirements is the most common
cause of project failure."

Been known for decades, if not centuries.

------
thisisit
While we might want to talk about fallacies of a government contract, the
reality is, well even software companies fall for it. Without giving much
away, my ex-employer a fairly large "product" company wanted to implement a
system from Peoplesoft's competitor.

The management wanted to save cost which meant neither full time employee
(FTE) were hired nor retrained. So they brought in IBM consultants.

Once the project was over IBM left only for the company to realize they had no
one to make incremental changes to the system or maintain it.

They spent extra money to hire another developer and consultancy to maintain
the system.

Takeaways: a. This was not the first time IBM had delivered shoddy product but
"low cost" was a driving factor so they get hired again and again. This is
true for both private companies and government.

b. Low cost also means companies don't want to hire FTEs rather make do with
contractors. This creates dependency.

c. It is not entirely on IBM or other consultancies but general ineptness too.
People holding on to their jobs, hiring and creating messes:

During the development of the aforementioned project one of the workflows fell
within my team's domain. But the finance team had some "developers" who wanted
to justify their salaries. They wanted to develop but had zero understanding
of what do words like "standards" and "best practices" meant. I spent hours
and hours arguing over why certain things should be done in a certain way. I
was always told I was too uptight.

In the end of module written by these "developers":

a. Transmits salary for the whole company via plain text files

b. If a, was not enough the text file is emailed to 60-70 people in the
company who have no authorization to look at salary data. It is always
maintained that without sufficient background people might not understand the
salary data at all because it uses "employee ids" instead of names.

So not everything was IBM's fault on the project.

Currently, the salary in text is a huge security risk but no one cares. All
our efforts to write a program to obfuscate data was rejected for being "too
complex".

------
darkstar999
> The company was told to use "off-the-shelf" software called PeopleSoft

Why didn't they just hire Oracle?

~~~
S_A_P
Oracle and other vendors rarely interfere when shop brings them a deal for
license revenue. If this was truly a 1 bid contract, it means that IBM likely
made the sale and software selection. At that point they brought this deal to
Oracle who may well have sent resources, but did not "run" the project. This
is another aspect to this sort of large project. Vendors are usually hesitant
to go all in with the implementation process, as it looks bad if too much
revenue comes from consulting on their balance sheet vs licenses.

If they derive most of their revenue from consulting and not licenses, it
gives the impression that its both difficult and expensive to implement. I am
willing to bet that a bullet point in the sales process is that it is easy and
cheap to implement...

------
alloowishus
You would think somewhere in the contract it would specify that the payroll
software, upon launching, most actually enable you to pay employees. I
understand why IBM would make sure the government had some responsibility, but
doesn't IBM have a responsibility to deliver something that actually works?
Well, apparently not, and I'm sure if that WAS stipulated in the contract,
they would never sign it. I have seen to the other side of the coin, I worked
for a consulting web shop years ago where they created a bad contract worth
70K, which was at the time their biggest contract. Yay, high fives all around
for the sales team. Unfortunately they ended up spending over 300k to get the
system to actually work because the contract was so vague and the requirements
so complex.

I smell some kind of corrupt kick back shinanegans going on here, I bet you
somebody in the government got a new sports car from IBM to get that contract,
and to have an endless maintenance part added in (which is where they make
their money).

------
lph
Mind boggling. What could possibly be so complicated about a payroll system?
Does anyone here have insight into this problem space?

~~~
teh_klev
Not completely definitive:

[http://wiki.c2.com/?WhyIsPayrollHard](http://wiki.c2.com/?WhyIsPayrollHard)

And that's just scratching the surface of the types of problems encountered.

Statutory compliance makes payroll software complex. There can be many
hundreds of rules defined by government and law (and combinations of these
rules) that can affect how much money ends up in your pay packet. In addition
to this the payroll software and the data held within needs to be continually
kept up to date and compliant with any new rule, change of rule or removal of
rules. That alone can mean many monthly updates just to keep in step with
legislative changes.

~~~
signal11
Some day laws will be written as s-expressions to solve just this problem.
(hat tip: Charlie Stross, _Accelerando_)

~~~
arethuza
I'm not sure that giving legislators macros is a good idea... (let alone
reader macros)....

------
equalunique
It's insanity to expect not to waste money when asking _IBM_ to install an
_Oracle_ CRM/SCM/FMS/HRMS/EPM/ERP suite. God knows it's running on some huge
SPARC or Power system with terabytes of RAM and thousands of threads. Maybe a
mainframe too. It's the worse possible combination. It's practically taking
the cost of IBM consultancy and the cost of Oracle software and multiplying
both. Suicide.

------
jlebrech
where does someone find government contracts to bid on? you could be like the
guys from War Dogs but instead of guns and ammo you trade code.

How does anyone knows there's something to bid on or do they only tell IBM? In
that case it's hardly IBM's fault is it.

~~~
Retric
The government creates a Request for Proposal (RFP) listing what the
government wants.

Companies have some listed period that could be as short as 7 days or as long
as 6 months to respond with a Proposal of work including costs.

Government then reviews and picks one based on the stated criteria.

This might seem like a reasonable solution, but it adds significant overhead.

~~~
gonzo41
So adding onto this, if you do a quick google for an exmaple of a government
RFP. OMFG! no agile in sight. You'll be greeted with about a thousand pages of
spec.

This is where the big shops earn their money. They know how to grind through
the paperwork.

~~~
Retric
You find plenty of Agile in government contracting. Generally the requirements
are very high level and the implementation details are after the fact.

I am also aware of some huge pivots where some minor feature became the only
thing the customer cared about.

------
jasonmaydie
The same Canadian goverment that decided to transfer all their IT systems to
one vendor Shared Services Canada which turned out to be another failed system
that almost every department including statistics canada blame for delays and
failures.

------
mbrodersen
The business model of most (all?) large scale software developers is to sell
the client the moon, at low cost, and then step by step make the client pay
more and more for the system, until the client ends up with a very expensive,
very hard to maintain system that kinda work, but requires ongoing and
expensive "maintenance" work that only the large scale developer can do. It is
a very lucrative business strategy. And they get away with it, because of the
client decision makers often not being competent enough to make decisions
about software systems development.

------
OliverJones
Didn't the same company recenly foul up a contract with another sovereign
government in Sweden? [http://www.politico.eu/article/sweden-data-breach-
privacy-se...](http://www.politico.eu/article/sweden-data-breach-privacy-
security-stefan-lofven/)

------
caffodian
I hope that the USDS-inspired CDS can prevent these situations in the future,
and improve the procurement requirements.

[https://digital.canada.ca/](https://digital.canada.ca/)

------
eein
I want an install of wordpress for 500 dollars, i'll handle the hosting costs
dont worry

\---

oh can i have a special blog template with x

oh can i have a store as well, i'll pay any associated licensing fees.

oh can i add a new payment processor

oh can i ...

why did you make me spend so much on this. I'll report to the media how awful
you are.

Said nobody but government.

------
ninjakeyboard
Nice work guys.

------
emersonrsantos
Why this doesn’t seem as a technological failure but a political struggle? I
can’t understand what is wrong with the system, but definitely see what’s
wrong with these public servants and unions.

------
eein
I want an install of wordpress for 500 dollars, i'll handle the hosting costs
dont worry

\---

oh can i have a special blog template with x

oh can i have a store as well, i'll pay any associated licensing fees.

oh can i have x plugin

oh can i have another x plugin

oh can i add a new payment processor

oh can i add bitbcoin

oh can i add some no-value features.

why did you make me spend so much on this. I'll report to the media how awful
you are.

Said nobody but government. (just the above bit)

