
A $300M IT flop - jsvine
http://fcw.com/articles/2014/07/24/it-flop-at-ssa.aspx
======
jeremymcanally
You'd have leaner, smarter firms that could actually deliver something getting
involved if the bidding process wasn't so obtuse and opaque.

When I was consulting, I responded to a few RFP's but they were always either
(a) obviously tailored to the capabilities of a single firm so there was no
way to win unless you were them or (b) so ridiculously vague or out of this
world impossible ("We need you to basically rewrite GitHub for us and the
budget is $30k") that it wasn't even worth the time. Even now where I work, we
have to deal with government agencies literally telling us they can't tell us
any requirements at all because that would give us an advantage over other
firms somehow. We just have to pitch our product to them and hope that it
fits, even though at this point in our development cycle we could (and really,
want to) make adjustments, feature additions, etc. to fit them if we were
allowed to.

Government contracting in IT, from my point of view, is nearly 99% about
"playing the game" and about 1% actually delivering something. Especially
since it seems that even if you fail spectacularly, you still get paid
(possibly paid even more to fix your stupidity).

~~~
pc86
I worked for a company who provided a SaaS to a few private entities but also
some very large state agencies, including an entire branch of the government.
I remember sitting in a meeting with my boss and the owner of the company,
while we _sat on the phone_ with someone from the procurement department of
another agency while they crafted the bid to only allow us to compete. It was
disgusting and they did nothing to hide it. The state employee (who knew the
company owner socially) even mentioned the competition by name, along the
lines of "if we require ______ then ______ won't be able to bid, right?"

~~~
beering
Competitors _can_ bid, but will have generally worse answers for the specially
crafted requirements. The competitors can either say "no, we don't do that"
and hope that the committee doesn't mind, say "yes, we could do that" and see
if the agency will pay extra for that special req, or say "yes, we do that"
and build out that feature if they win the bid.

~~~
danielweber
Eh, once i was at a startup where we were selling our mostly COTS software to
a government agency, but they had to put it out for a bid. They basically
lifted our marketing material into the req and said "software must do all <X>
and cost $<Y>."

We had a competitor who had copied a bunch of our stuff, including our typos,
but they were missing one essential feature. Yet they still got the bid
somehow. A mystery for the ages.

------
rayiner
Interesting quote from the article:

> 'You have companies like Lockheed Martin -- they're not web development
> companies, they don't do work in the private sector, they're not building
> start-ups,' Gershman said. 'There's very little in the market that forces
> them to adopt new technologies.'

All of these are true, but I think this glosses over some fundamental
structural problems with government contracting. The SSA has almost 60 million
beneficiaries, and deals with distributing an entitlement that is extremely
sensitive and protected by layers of legal process. 8,000 Social Security
checks get lost,[1] and its not a blog post, its national news.

Moreover, a lot of lean and agile processes just aren't consistent with
political reality. There's no minimal viable product, there's no invite-only
beta testing, there's no iteration, there's no A/B testing, etc. Technical
decisions ("let's just roll out to northern Alabama first") become divisive
political ones. Purchasing decisions ("we've had good luck with this vendor
before") become the basis for accusations of favoritism and corruption.

Perhaps most importantly, there's no failing fast. The private sector accepts
the fact that many engineering projects will fail.[2] Good companies take care
not to punish failure unreasonably. But in the public sector, failure is just
license for those opposed to project to crow: "we shouldn't have been doing
this anyway!" If you fail fast, it's easy to pin the blame on you. This
creates a perverse incentive to drag a project along so failure happens over
years if not decades.

[1] [https://medium.com/@jan.curn/how-bug-in-dropbox-
permanently-...](https://medium.com/@jan.curn/how-bug-in-dropbox-permanently-
deleted-my-8000-photos-cb7dcf13647b)

[2] Most big IT projects fail:
[http://www.computerworld.com/s/article/9243396/Healthcare.go...](http://www.computerworld.com/s/article/9243396/Healthcare.gov_website_didn_t_have_a_chance_in_hell_).
Most startups fail. Most internal prototypes never amount to anything. This is
just how engineering works.

~~~
enraged_camel
>>Perhaps most importantly, there's no failing fast. The private sector
accepts the fact that many engineering projects will fail.[2] Good companies
take care not to punish failure unreasonably. But in the public sector,
failure is just license for those opposed to project to crow: "we shouldn't
have been doing this anyway!" If you fail fast, it's easy to pin the blame on
you.

This is not unreasonable. After all, projects in the private sector are funded
by private funds. If you fail, you only waste your own money. Whereas public
projects are funded by the taxpayer. When they fail, what's really happening
is that taxpayer money is going into the pockets of some private company, but
there's only a failed project to show for it. So yeah, of course the backlash
is going to be much stronger.

~~~
timdierks
In the private sector, investors provide capital that companies (and employees
thereof) use to pursue projects. When the company fails, the employees are
generally still paid (while some companies have employees who participate in
profits / share risk with investors, it's rare for it to be a dominant form of
compensation). Thus, the situation is more similar than you describe: if the
projects fail, the investor/owner has paid out a lot of money to employees and
vendors and is left with nothing.

The difference is that smart investors/owners are aware that the possibility
of failure is inextricably tied to any attempt to excel, and allows people to
try things that might fail in the pursuit of success.

In our current political climate (and probably in other climates as well),
governments are dumber, and tend to drive their staff to manage towards over-
safety and covering of asses rather than towards best outcomes.

------
8102460
I have actually worked on the DCPS product as a former LM employee, so here's
what I saw: there are two sides to the problem. The biggest problem is the
number of stakeholders. Currently, each state has their own system to process
disability cases, and each system has its own way of doing things. SSA &
Lockheed are trying to consolidate these 50+ systems into one national system.
Every state this product is rolled out in creates some new challenge to make
it work the way the state's SSA office expects. Hopefully this new exec will
have the power to make some decisions and say "no" once in awhile.

Lockheed has their own set of problems, mostly hinging on the contract they've
signed with SSA. They are forced to hire a certain number of subs per their
contract - hiring a percentage of subs is not unusual, but the level in this
contract is very high. As such, the subs know they can force feed trash into
LM. The resumes coming from the subs were complete bullshit. Things like "10
years of experience administering X", but then in the interview, you find out
they know next to nothing. Especially bad with databases when you even try to
ask them how to write a join. One meeting we had while I was still there was a
"skills assessment" \- from 1 to 10, how well do you know X? There was no
negative to this meeting, we just wanted to know who knew what. The number of
"10s" in that meeting was beyond belief.

So sure, Lockheed may not have a lot of experience here, but that problem is
exacerbated by the stakeholders and the contractors they are forced to hire.

~~~
nazgul
Why are the forced to hire subcontractors, and a certain # of them? Do you
know what the thinking is behind that?

~~~
8102460
The overall idea behind being "forced" to hire subs is that it helps promote
smaller, local businesses. In the past this worked out very well, because the
subs brought in some very knowledgeable people. I believe the old number was
25% of the people, and you could pull from anywhere. If you had a 20 person
team, you could have 15 LM, 2 from Company X, and 3 from Company Y.

On DCPS, the problem is that LM is forced to hire 45% (FORTY-FIVE) of the subs
through _one_ specific subcontractor. That company had a previous contract
with SSA, and SSA liked them. Since the sub essentially lost their work with
SSA to LM, and they know LM was contractually obligated to hire through them,
they stuffed us with a ton of below average folks.

On the face of it, I agree with the idea of a large contractor having to sub
out some of the work to the smaller players; it's just that LM overplayed
their hand trying to get this contract, and got screwed for it. But that's
another story on how LM management is trying to make revenue numbers for
bonuses...

------
adwf
It's still got nothing on the NHS £10bn IT flop...

I always wonder when I hear about these projects, where exactly has the money
gone? It can't possibly be in hardware, that would make some of the biggest
data centres in the world just for the NHS. I can't even see how it can be
manpower, how many IT people can you get for £10bn? And surely if it were
corruption, someone would notice so much money being squirreled away...

There's just no logical explanation behind these things.

~~~
martiuk
gov.uk which had a great success, was done by hiring their own people to work
on it. I don't see why anyone would think outsourcing is good for public
projects.

~~~
beat
I worked on a high profile government project that had five different vendors
involved. FIVE. The inter-organizational management problems were astonishing.
I'm pretty sure that by the time I left, managers actually outnumbered
technical staff.

The biggest problem, though, was the lack of pushback. The government
officials would ask for the moon, and the vendors would promise to deliver it.
There was almost no one saying NO - the word that project most desperately
needed to hear. Features outweighed core infrastructure reliability, so
configuration management was a nightmare.

------
incision
_> 'The project was conceived to replace 54 separate components used in the
SSA's disability determination system.'_

This is the bit that jumps out at me as clear indicator of the sort
dysfunction that dooms projects like this more so than anything else.

It's indicative of a the classic exploding scope and fragmented
responsibilities leading to an overall lack of accountability that
unfortunately characterizes IT in and around Government.

Gershman seems to agree.

 _" If there's not strong leadership, problems arise with coordinating work
between contractors, and there's no one person looking at the system as a
single entity -- everyone is making sure their piece is done, and fulfilling
their responsibility up to a certain point,"_

However, I'm not wholly keen on the the start-ups as a solution direction
taken later. It's not that it's wrong, just beside the point unless the start-
up savior is specifically targetting culture and process change as opposed to
technical implementations.

The 54-armed kraken of a defined by committee project will drown and devour a
fleet of confused start-ups lost in the mist of *BE requirements and contract
mods just as readily as the Lockheed leviathan.

~~~
greggersh
What I meant by startups is that in that world, there's market forces driving
you to do things better, faster and cheaper. Not necc. that we need startups
to solve the IT problems of government (although that could help), but that we
need to create an environment where there's market forces pushing government
contractors to do things for less people/less money and better results (which
are all very, very possible).

------
ryandvm
This is business as usual in government contracting. The goal is not really to
deliver working software - instead, it's multiple companies working together
to figure out how they can maximally extract income from an entity with
unfathomably deep pockets. It's layer upon layer of red tape sprinkled with
the occasional sociopath just to spice things up.

The filthiest part is that nobody but the taxpayer is interested in things
being more efficient. The government agencies don't really care because there
is no real financial accountability. Besides, if they use up all their
budgets, they just get to ask for more next year. And contractors generally
have their profits limited to a percentage of the project cost, so their goal
is to balloon the cost as large as possible.

For several years I worked on a very large government project. When you're
sitting in a two-hour, 100 person, sprint review every two weeks and everyone
is fucking off on their phones the whole time, it literally makes you question
your sanity.

------
discardorama
The sad thing is not that the government pissed away $300M on this project;
but that the prime, LMFS, will continue to get away with it because there are
no penalties!

But the system is rigged that way: if you do decide to hold them accountable,
you'll find that they have documented every little specification change that
the SSA put in after the project was awarded. So now the blame comes back to
the SSA. Both sides will point fingers at each other; the consultants on this
project will laugh (on their way to the bank), and things will remain the
same.

In other words: the system of contracting in place today is broken. From
labyrinthine requirements to bid for one, to obtuse specifications to lack of
oversight; it's one huge mess.

And the system is corrupt to boot. Primes like LMFS, etc. employ former
employees of these agencies (and vice versa), so it's very hard for an
outsider to break in.

------
easytiger
Pennies in an ocean:

[http://www.theguardian.com/commentisfree/2011/aug/03/nhs-
dat...](http://www.theguardian.com/commentisfree/2011/aug/03/nhs-database-
digital-disaster)

> ...final cost to be as high as £20bn, indicating a cost overrun of 440% to
> 770%

------
mathattack
"For past 5 years, Release 1.0 is consistently projected to be 24-32 months
away."

I have worked on project like this. Usually it's the sign of management
failure (poor communication, poorly defined requirements, bad governance,
inability to retain knowledgable employees) rather than technical feasibility.
It's not like the software is pushing technical boundaries.

"Echoing McKinsey, Gershman said that without a strong leadership element, IT
projects of this magnitude are almost doomed to fail from the start."

This is a big problem. Because employees know these problems will fail, the
smartest avoid them like the plague.

------
remon
It's amazing to me how consistently government IT projects fail and how
widespread the problem is. A simple google teaches us that this has been and
still is a consistent problem for quite a lot of countries. Some with an
incredibly strong tech sector (the US is obviously a prime example). Surely
there must be a way to organize/orchestrate government IT projects in a way
that improves delivery and risk management. And...300 million USD?

~~~
al2o3cr
Scratch out "government" above - the overall failure rate of IT projects in
_general_ is crazy. The difference is that private project failures don't get
news coverage (barring utter disasters that bankrupt the company).

------
bane
For more like this

[https://en.wikipedia.org/wiki/Virtual_Case_File](https://en.wikipedia.org/wiki/Virtual_Case_File)

[http://www.reuters.com/article/2012/03/14/us-manhattan-
saic-...](http://www.reuters.com/article/2012/03/14/us-manhattan-saic-
idUSBRE82D14B20120314)

------
androtheos
No wonder taxes are so unbelievably high. The FBI did the same thing a few
years ago.
[http://www.cnn.com/2005/US/02/03/fbi.computers/](http://www.cnn.com/2005/US/02/03/fbi.computers/)

~~~
duaneb
Yes, it's not at all programs like the absolutely-crucial F-35 program costing
$399B up front.

------
steven2012
Feeding from the government trough is where people REALLY get rich. This is
why Palantir is pretty successful, because getting in bed with the government
means being able to feed off our tax money much easier at elevated prices.

~~~
duaneb
> This is why Palantir is pretty successful

Ahhhh

------
junto
McKinsey and Co probably charged the government a couple of million dollars to
get some of its junior partners to write that report which eventually a senior
partner signed off before golf and took the credit.

Says the cynic in me...

------
jsvine
> Throughout the past six years, the project has been stuck in the beta phase.
> According to McKinsey, "for past 5 years, Release 1.0 is consistently
> projected to be 24-32 months away."

------
Xophmeister
Rather offtopic, but does anyone else find the dollar glyph in the header font
jarring?

~~~
quux
Now that you mention it, yeah it is... It looks like a lot of the symbols and
punctuation characters in Garamond Pro are slanted with no logical pattern or
reason. Would drive me nuts if I were using it.

[http://i.imgur.com/Cbd74Gq.png](http://i.imgur.com/Cbd74Gq.png)

