
List of failed and over-budget custom software projects - monort
https://en.wikipedia.org/wiki/List_of_failed_and_overbudget_custom_software_projects
======
strommen
In 2006 I worked briefly on an air-traffic control system (ERAM) that was
eventually delivered 5 years late and $400M over budget. I was a small cog in
a giant government-contract-software machine, but it was easy to see the
project was going to be a disaster.

There are some pretty basic strategies to make building "large" software
easier. All of them were attempted, but in an entirely broken way:

\- Break the architecture down into smaller components _that can be tested
independently_.

\- Break the functionality down into smaller features _that can be released
independently_.

\- Develop automated tests _at the system level_ (i.e. integration tests, not
unit tests).

The software took 2-4 minutes to launch in development, and very few of us
knew how to meaningfully test it. This meant that any change that compiled
(and passed code review) would ship.

Oh yeah - and the system used binary structures in shared memory to
communicate across processes, which were running code written in two entirely
different languages.

~~~
dotnetchris
> the system used binary structures in shared memory to communicate across
> processes, which were running code written in two entirely different
> languages.

What could go wrong

~~~
Someone1234
"We should have just used XML for everything!" \- said every Java developer
ever.

"We should just have used binary for everything!" \- said every C/C++
developer ever.

"We should just have used Json for everything!" \- said everyone currently.

~~~
fratlas
Any reason not to use JSON? Apart from needing lightweight traffic, in which
case MQTT would be fine, right?

~~~
com2kid
* You need strict typing * You need well defined numeric formats * Message volume is high enough that the parsing costs of messages outweigh the savings on implementation time * You cannot guarantee plain text messages won't be managed in transit.

You can get around the strict typing by adopting a schema format, but then you
have a tooling problem that all the languages you need to support may not
support your chosen JSON schema format.

Other solutions such as protobufs exist and are far better for transmitting
large amounts of binary data.

------
edgyswingset
I remember, when as a resident of Oregon, I was wondering why we couldn't sign
up with the national healthcare system. I then noticed that the Cover Oregon
system was being built by Oracle.

Suffice to say, I wasn't the least bit surprised that it was a complete and
total catastrophe, and even less surprised that Oracle tried to sue Oregon for
attempting to use the software. Nothing like LDD - Lawyer Driven Development.

~~~
Yhippa
I remember this being quite the story. Anybody know what came of it? Still in
litigation?

~~~
schmichael
Still in litigation. OR has spent more than $6.6m in legal fees fighting
Oracle.

~~~
killface
I still don't understand why people trust Oracle consultants. Their list of
failures is ten miles long. I guess nobody gets fired for hiring Oracle.

~~~
tdeck
Oracle is today's IBM, just as IBM is today's Infosys.

------
kozukumi
The modern NHS system in the UK was the biggest fuck up I have ever seen. I
worked with people who also worked on the project. They are seriously amazing
people but apparently all the layers of management, red tape and politics was,
according to my friends, the worst they have ever experienced. Two friends
were so traumatised by the experience they ended up taking 2 years off work to
travel/recover.

The sad thing is it shouldn't have been such a difficult thing to develop.
Complex sure but no more complex than many other similar systems. It was
destroyed by too much meddling from the government adding, removing and
changing the spec every other day and many, many managers just finding non-
sense jobs that were totally pointless.

It is shocking it went so over budget and was allowed to continue doing so for
so, so long.

~~~
teh_klev
Which NHS body in the UK are you talking about? There are four separate
healthcare services in the UK which operate and are funded separately:

NHS Scotland

NHS England

NHS Wales

HSC Northern Ireland

~~~
noir_lord
NHS England - The National Programme for IT.

It is as far as I've seen the 'greatest' fuckup in UK development history,
12.something billion down the drain, massive overspending, they basically made
every mistake you could make short of setting the office on fire.

~~~
aledalgrande
They should make mismanagement of a government project become a felony.

~~~
dragonwriter
> They should make mismanagement of a government project become a felony.

This would involve either:

(1) Clear and unambiguous definition of "mismanagement" involving distant-
from-the-specific-project procedural mandates, which would reduce
"mismanagement" by the definition applied but increase (and mandate) what
would be functionally mismanagement for specific real projects (lots of
government regulation designed to prevent mismanagement all around the world,
while usually not specifically felony criminal rules, works this way now), or

(2) Being so ambiguous and vague so as to provide a basis for arbitrary
prosecution, such that it would make sure no one wanted to touch management of
a government project at all (at least -- assuming this was the US or one with
similar fundamental rules as to what can be an enforceable criminal law --
until it was inevitably ruled unenforceable as impermissibly vague.)

~~~
aledalgrande
It would indeed be a hard task to define what is "mismanagement", but spending
billions of tax money earned by hard workers all around the country on
salaries of incompetent people and fees from incompetent consultancies has at
least to be slowed down, if not stopped. As civic responsibility is not
enough, a higher level of control is needed. I am open to other solutions.

~~~
dragonwriter
> It would indeed be a hard task to define what is "mismanagement", but
> spending billions of tax money earned by hard workers all around the country
> on salaries of incompetent people and fees from incompetent consultancies
> has at least to be slowed down, if not stopped. As civic responsibility is
> not enough, a higher level of control is needed.

Much of that waste is a product of the "higher level of control" adopted, in
law and policy governing government IT work, in response to previous failures,
which has mandated additional bureaucratic process, and causes more and more
decisions to be made farther and farther from the people with either the
specific business knowledge or the specific technical knowledge of the project
being executed as more layers of "higher levels of control" are implemented.

I am not really sanguine about the prospects of more of the approach that has
made things worse suddenly making things better instead.

------
mrweasel
I should sit down and add the failed IT projects of the Danish government, but
I don't have that kind of time. Our government is notorious for initiating
software projects that ends in almost predictable failure.

The latest project to fail was a system for the Danish Tax Office. It was
suppose to automatically collect debt owed by citizens and companies.It's not
entirely certain how that was suppose to work and it has been described as a
utopia project with an unprecedented expectation of what kind of problem
computers are capable of handling. The specification where 9000 pages long,
tax law not included, and some how failing to supply a sufficient amount of
actual requirements. It failed, of cause, costing almost $150.000.000 (DKK
1.000.000.0000), not including the 78 Billion Danish Kroner the tax office
haven't been able to collect due the system not being delivered on time and
generally not working.

~~~
busterarm
Doesn't the government there (and most companies for that matter) contract IBM
for big software projects?

You know the old adage "nobody gets fired for buying IBM".

I have a friend that's partner at a consultancy in Copenhagen and he says IBM
is often their competition.

~~~
mrweasel
Mostly CSC actually. They have contract for systems that should have been part
of licensing rounds, which would have allowed other to take over support, but
no other where ever allowed to bid. The reason: CSC documented the systems so
poorly that no one else will be able to handle the support.

The government knows that it's illegal, yet CSCs contracts have automatically
been renewed, multiple times.

But yes, IBM have a large number of contracts as well. CSC just seems to be
behind the worst projects. I suspect they bid for project and just focus on
delivering exactly what's in the contract, knowing that it will never work.
Either that or they are even more incompetent that anyone could imagine.

~~~
busterarm
Given my experience with the managers of CSC here in the States, that sounds
exactly like what I'd expect.

From my discussions with that friend I mentioned though it sounds like the
attitude is changing and there's more work going to smaller shops like his
(though friendships with government officials seems to help loads when it
comes to securing work). Honestly it sounds like a really exciting environment
to work in when we talk about it and I wish I had the time to really learn
Danish so I could have a decent chance of securing good work out there.

------
orf
Can someone explain to me how the fuck my government can spend _15.8bn_ [1] on
a software project? We could build 5 Queen Elizabeth class aircraft carriers
for that. Five! For the cost of a bloated Java app!

Seriously, someone fill me in please. How can a software project possibly ever
cost more than a billion pounds? You could employ 20,000 software engineers
full time for £50K ($70k) a year for that much, and I expect the number of
engineers working on Universal Credit is much lower than that and paid less.

1\.
[https://en.wikipedia.org/wiki/Universal_Credit#Costs](https://en.wikipedia.org/wiki/Universal_Credit#Costs)

~~~
detaro
They don't employ the developers. The development is done by companies that
might pay the developers that much, but bill the client more per hour and bill
the client more time. And of course you can't forget project managers,
architects and other consultants that are more expensive. And have a lot of
meetings. And hardware, and oracle databases/SAP licenses, and support
contracts for those, and audits, and extra charges for change requests, and
travel budgets, and...

Just because one team of developers could do something in 200 man hours
doesn't mean you can't spend and bill 2000 for it.

And that's just development. Data migrations and user training/support are
going to be massive issues if you replace or newly introduce a widely used
system.

~~~
orf
> Just because one team of developers could do something in 200 man hours
> doesn't mean you can't spend and bill 2000 for it.

It doesn't mean you _should_ pay, or plan, to spend 20,000 man hours in
meetings and devote huge chunks of your budget to expensive licenses for
something that could take 200 man hours, especially when it's taxpayers money.

Sure there are unavoidable overheads but why does our government keep
splashing __billions __on them and failing, with little accountability?

Training is going to be expensive, sure, but perhaps plow millions of pounds
into that after the program is finished. Maybe don't use the same massive,
costly consultancies that fail to deliver? The Obamacare website was a great
example of this IMO. Costly consultancy failed to deliver, small agile teams
brought in to save the day - if only a bit too late.

~~~
Lawtonfogle
Basically all the laws that are written to avoid government abuse and
corruption create a corrupted system of regulatory capture.

Here is the simplest example. Say you need a device that does X. Now, I can go
down to the local hardware store and pick up a device that does X (and Y and
Z) for $40. But the law is written so that I can't just buy it where I want,
because I might be wasteful and I might buy it from my buddy. So instead, I
have to put out a bid that says I want a device that can do X and only select
whom ever wins a bid. But there are a lot of regulations in who can bid, so
the local hardware store doesn't ever even try. So I say I want a device that
does X, and I end up getting a couple offers, the best of which is a device
that barely does X for $200. That is the best offer I receive, so I have to
take it. So now I have a much worse device at a much higher cost. Now,
multiply this with every single thing you need to pay for in a software
project (including other software, licenses, hardware, manpower, even the
paper to print specs on).

~~~
petra
>> the best of which is a device that barely does X for $200.

OK, after we did the bid, and that's the result, why can't i just go buy
something if it's less than half the price ? even if i buy it from my buddy,
as long as it fits ?

~~~
jjoonathan
Because your buddy was secretly cutting corners using Chinese chips that have
a backdoor. Or chips that weren't bonded to spec and last only 2 months in the
desert heat.

That's the theory anyway.

------
jamessb
IEEE Spectrum had a great series of interactive information graphics on
'Lessons from a decade of IT failures' [0].

In particular, "The Staggering Impact of IT Systems Gone Wrong" [1] and
"Monuments to failure" [2] are good lists of failures, many of which are not
included on Wikipedia's list.

[0]: [http://spectrum.ieee.org/static/lessons-from-a-decade-of-
it-...](http://spectrum.ieee.org/static/lessons-from-a-decade-of-it-failures)

[1]: [http://spectrum.ieee.org/static/the-staggering-impact-of-
it-...](http://spectrum.ieee.org/static/the-staggering-impact-of-it-systems-
gone-wrong)

[2]: [http://spectrum.ieee.org/static/monuments-to-
failure](http://spectrum.ieee.org/static/monuments-to-failure)

~~~
thrownaway2424
I don't know if those are all in the same category. For example the Army
cancelled Future Combat System because it didn't work. Would the project have
been considered a "success" if they had put it into production instead?

For my money, a real failure is like the giant BART fiasco in the 90s where
they spent 80 million dollars on a new train control system that simply never
existed, at all. They had to sue General Electric to get their money back.

~~~
AnimalMuppet
> I don't know if those are all in the same category. For example the Army
> cancelled Future Combat System because it didn't work. Would the project
> have been considered a "success" if they had put it into production instead?

If it didn't work? Hopefully not.

But military stuff is a bit different. Military systems tend to try to push
the state of the art as far as possible, for as much edge in combat as
possible. That makes for much higher chances of failure.

------
ergothus
The primary problem with lists like this is that they don't compare to
successes.

For example:

* I've been on projects where we're rewritten applications from the ground up that were totally successful and we were better able to build in the future because we had a clean setup.

* I've seen government IT shops hire contractors to do a large project and ended up with something better than the internal devs would have done.

Both of these are situations where the opposite case (time-consuming and
expensive failure) gets a lot of attention. (and, in my experience, failure is
more common). Show me one of those failures and my instinct is to say "Of
COURSE that was a failure! It was obviously going to fail!". But, in reality,
not always.

Rather than building unhelpful conventional wisdom, we should be trying to
understand the distinctions between the failures and the successes.

------
tyre
96% of government IT procurements over $10m are deemed failures.[1]

Absolutely shocking how ineffective the system of contractors and RFPs is.
Part of it is a mindset, part of it is just needing better options.

[1]
[https://www.washingtonpost.com/news/wonk/wp/2013/10/22/the-l...](https://www.washingtonpost.com/news/wonk/wp/2013/10/22/the-
lessons-of-healthcare-gov-stretch-far-beyond-obamacare/)

------
pkorzeniewski
When it comes to government projects, I always wonder how many people involved
treat it only as an opportunity to make as much money as possible without
being caught - after several years you just sweep the whole thing under carpet
so nobody remembers how much money has been wasted and you're set.

~~~
zhte415
This is true not only of government projects, but of all projects.

Anyone in charge of a budget is potentially corrupt. I'd argue that it is more
common in corporates than in government, because the personal damage that
could be inflicted is less. A company doesn't want reputation damaged by being
involved in fraudulent bidding (so will sweep under the carpet readily), and
as long as the job gets done, nobody cares (unless internal politics get
stirred). The employee may be fired, but they can go somewhere else. And all
is off the public record.

------
mcguire
These lists seem a little short. What about:

* The Distributed Computing Environment (or at least the OS/2 portion of it, anyway), [https://en.wikipedia.org/wiki/Distributed_Computing_Environm...](https://en.wikipedia.org/wiki/Distributed_Computing_Environment)

* Taligent, [https://en.wikipedia.org/wiki/Taligent](https://en.wikipedia.org/wiki/Taligent)

* IBM Workplace OS, [https://en.wikipedia.org/wiki/IBM_Workplace_OS](https://en.wikipedia.org/wiki/IBM_Workplace_OS)

And that's just a few from off my personal resume.

------
cthulhujr
>This article contains one or more incomplete lists which may never be able to
satisfy particular standards for completeness

Well isn't that ironic.

~~~
SmellyGeekBoy
Don't you think?

------
pessimizer
SAP doesn't even get a mention?

Waste Management: [http://www.computerworld.com/article/2517917/enterprise-
appl...](http://www.computerworld.com/article/2517917/enterprise-
applications/sap--waste-management-settle-lawsuit.html)

Levi's:
[http://www.theregister.co.uk/2008/07/10/levis_erp_costs/](http://www.theregister.co.uk/2008/07/10/levis_erp_costs/)

~~~
mschuster91
The problem with SAP isn't the software, it's "consultants" writing
purposefully bad code - the customer can't assess the delivered quality and
the consultant will keep his "ongoing maintenance" fee.

(Okay, yes, the software itself is a problem too but it's not the major
problem)

------
OhHeyItsE
Would have been much easier to create a list of projects that were completed
within budget (assuming that's ever actually happened).

~~~
softawre
Hey, I wrote a little shell script yesterday that I had budgeted 30 minutes
for and had it done in 20.

But outside of that one thing.. lol

------
ap22213
Note: very few, if any, of these examples failed because someone on the team
didn't pass a coding test. Next time you interview someone remember: bad code
could bring your project down, but there are 9 other factors that are more
likely to bring your project down.

------
natetan
Great idea.

The UK tried a NOMS (offender management) system and that was an expensive
disaster too..

Would be nice to put down the contracting companies who were ripping of the
Govt along with the civil servants who over saw the massive waste of tax
payers money/time.

------
tyingq
Of course, there's a level of disclosure for Government run projects that
means their blunders get exposed more easily.

I'm quite sure there are some spectacular failures in the corporate world that
won't make the list.

~~~
DubiousPusher
There's also an asymmetry in the way we think about public and private
projects.

When a piece of private software flops, it still exists as a product and
however bad, often offsets some of its cost in sales.

Since most public software is offered for free, no cost is visibly recouped.

Healthcare.gov is actually a great example. It was a colossal failure at
first. A quantity easily measured in dollars spent. However, it has largely
functioned in the years since but measuring its positive economic impact is
hard because the service is delivered free. None if that money has been
reclaimed but it certainly has had some positive economic value.

~~~
bdavisx
No, they don't always flop and still exist privately - sometimes they are bad
enough that they don't ever see the light of day anywhere - even internally to
the company.

------
frik
Now we need a list of overbudget customized COTS software projects.

With all these companies running on Excel macros, years old SAP R/2 or 3 (just
two common examples) and so on, the list will be extensive.

(COTS = commercial off-the-shelf)

~~~
rbanffy
That'd need a whole Wikipedia just for it.

------
codingdave
That list is much short than the headline would imply. I was afraid of a list
of thousand of projects, with a final footnote of "Every Enterprise software
application ever written."

------
repeek
For those interested in this topic, I highly recommend episode 181 of the
Omega Tau podcast, "Why Megaprojects Fail (and what to do about it)" [1].

It's a fascinating listen into the psychology behind gigantic failures.

[1] [http://omegataupodcast.net/2015/09/181-why-megaprojects-
fail...](http://omegataupodcast.net/2015/09/181-why-megaprojects-fail-and-
what-to-do-about-it/)

------
amist
As a side note, I cannot understand why these articles are written in a table
format. Where there are cells with such a long description, bullets would be
much suitable.

~~~
organsnyder
Agreed on the formatting, but the sorting functionality can be handy.

------
rdudekul
I was trying to look for patterns in these projects. Looks like most of them
were outsourced and most of them were US/UK Government projects. It would be
interesting to see patterns in very successful US/UK Government projects.

My personal experience has been that it almost always boils down to upper
management/leadership issues, that result in hiring weak project managers
which in turn leads to talent drain on the implementation side.

~~~
gaius
What it boils down to is no-one is spending their own money, they have a
bottomless pit from the poor taxpayer.

------
mrslave
I will document my bias before reading the article in the hope that I am
proven wrong.

I estimate that most of these have occurred in apparatus of the state,
including organizations that superficially are not but for all practical
purposes (funding, license, regulation) are.

Wise man once say: there's nothing easier in this world than spending someone
else's money.

------
api
I see a lot of billions, with a b, and then I look at what small teams do on
bootstrapping or little seed rounds. The mind boggles.

~~~
jandrese
A huge, huge, cost driver in many of the applications is the fact that they
have to integrate with the old systems in numerous ways.

A startup could probably build the basic version of many of these apps no
problem, but that's only the tip of the iceberg. When you have to then import
millions of customer records, which were not stored in any particular format
(they're idiosyncratic with each doctor/office), and then integrate with the
COBOL based system vendor 1 uses, and the IBM Mainframe that vendor 2 uses,
oh, and those systems were never intended to interconnect with anything and
the vendors are years behind schedule as well, and all while you have lawyers
breathing down your neck with a 28 inch thick binder with all of the
regulations you need to follow written in impenetrable legalese you can see
how it can cost billions of dollars.

Your startup guys heads would explode if you asked them to write a formal
specification for an interconnection system between their elegant web
frameworks and a big blob binary COBOL based record system on an IBM 360 with
only hardcopy documentation written by a vendor that went out of business in
the 80s. And yes, healthcare systems still run in environments like that for
the reasons mentioned above: making a new system is too hard/expensive.

Oh, and the vendor that had the lowest bid for the contract is planning to
outsource to some country that doesn't speak English and is 12 hours offset
from your local time.

~~~
jonesb6
It still seems like an excuse, "it's hard". The cost of "hard" shouldn't be
$15B.

------
stretchwithme
The Software Engineering Radio podcast did an episode on this recently:

[http://www.se-radio.net/2016/02/se-radio-
epislode-250-jurgen...](http://www.se-radio.net/2016/02/se-radio-
epislode-250-jurgen-laartz-and-alexander-budzier-on-why-large-it-projects-
fail/)

------
jayvanguard
I'm surprised the outsourcing companies aren't listed on this. They are often
huge contributors to the overruns.

------
tetraverse
"Sabina SEIFERT's Project: Software Project Failures"

[http://www.scism.lsbu.ac.uk/inmandw/past/9697/projects/prj12...](http://www.scism.lsbu.ac.uk/inmandw/past/9697/projects/prj127.html)

------
panglott
Why are these all in the UK or US? Better transparency?

~~~
slantyyz
>> Why are these all in the UK or US? Better transparency?

Maybe because that particular Wikipedia page's editors are from the UK or US?
[edit: just speculating, don't know this for a fact]

If I recall correctly, the Gun Registry [1] in Canada was a huge, expensive
debacle, with a lot of the costs coming from the underestimation of the
software development.

[1]
[https://en.wikipedia.org/wiki/Canadian_Firearms_Registry](https://en.wikipedia.org/wiki/Canadian_Firearms_Registry)

------
DyslexicAtheist
As far as I can see all software built in Germany always ships on time and is
within budget. Or maybe the page needs an update :-)

------
timwaagh
this is why haskell has support for infinite lists

------
TheAndruu
This is by no means a conclusive list.

------
Grue3
...in UK and US. Apparently other countries never had any of these.

------
jordache
when they list healthcare.gov has costed 1.5bn, does that include setting up
the back office and staffing, or simply the project that responsible for
putting the website?

------
boxfire
See Also: Agile Software Development

------
peterburkimsher
Did nobody mention Xanadu?

