
Alaska government spent $28M on database project before it was scrapped - japhyr
http://www.adn.com/article/20150207/alaska-government-spent-nearly-28-million-database-project-it-was-scrapped
======
us0r
Ah, SAIC (now known as Leidos). When you think of shady, corrupt government
contractors - these guys are basically the arm pit of that lot.

I always wondered why they take on such small projects. Northrop just botched
a system in Tennessee[0].

[http://www.tennessean.com/story/news/2015/01/12/tenncare-
fir...](http://www.tennessean.com/story/news/2015/01/12/tenncare-fires-
northrop-grumman/21643813/)

~~~
spiritplumber
Because it's basically free money AND free on-the-job training/evaluation for
their junior guys.

------
japhyr
This kind of story always makes me think I'll never really be out of work. I
like to think that given this level of funding, I could do a much better job
in a situation like this than the firm that was hired did.

I've been programming recreationally since I was a kid in the '70s, and I've
started to do some professional work in the last few years. I started out in
BASIC, and then became familiar with C, Fortran, and Pascal in college. I've
since learned Java, and now my language of choice is Python. I have a basic
competence in SQL, git, and any number of other tools we use in modern
development approaches.

I know work like this isn't easy. I know the data is stored in esoteric
structures on long-outdated hardware, running on OSes I've never touched. I
know there are issues with security clearances and background checks. But I
have to think - give me $28m, let me work on this full time for a couple
years, and I'll have some meaningful work done and some usable end products.
I'd give myself a modest salary, and hire people with way more experience than
myself in the areas needed. I can even see a path into this line of work -
complete a series of successively more complex and sensitive data-migration
projects. I like to think that doing that work wouldn't turn me into someone
running a firm like the one hired in this situation.

For people who've worked on projects like this that turned out well, is this
something any of us with reasonable competence could get into, if we build an
appropriate portfolio?

Also, I live in Alaska so I find this story particularly frustrating. Not just
because of the wasted money, but also because I want our law enforcement
personnel to have a well-functioning system at their disposal.

~~~
x0x0
But the trick is, you probably couldn't do a better job.

There is little technical challenge in a product like this. Building crud apps
that scale to low tens of thousands of users is not that challenging. Since (I
believe) we all share that belief, then why do projects like this keep
failing?

I've worked on large medical information systems. The problems those encounter
are

(1) you essentially never get to bulk replace entire systems; you will have to
support ongoing data ingestion and export to other running systems, often
systems not designed to support this or with poor apis;

(1a) if / when you fuck up, critical processes are disrupted. You have to
engineer your systems and create backup processes to make this very unlikely
to happen. There is a case I can't really share too many details of where a
database was accidentally dropped in a datacenter of a hospital chain you've
heard of. Said hospital chain was unable to access medical records for 20+
hours; this was in the 90s running on giant hpux machines and it took a long
time to reimport data off tape. My employer was on the hook for $30M iirc, and
I wouldn't be surprised if this actually contributed to serious patient
injuries or deaths. Now obviously law enforcement isn't quite as critical as
medical records, but it's not far behind.

(2) over the life of such systems, lots of workflows / processes have
developed around using them inside hospitals or large doctors offices. You
have to discover these processes, mostly stored in the heads of line level
workers and their immediate supervisors, and figure out how to enable them in
your shiny new system. To be concrete: the medical record system didn't
support some requirement. That didn't stop users from needing it. So employees
or doctors decided to use a field in a certain way, and they know how to
interpret the presence and/or data in that field. You don't even know a field
is being repurposed, let alone how they actually use it. This will be a large
problem importing all the old data and in recreating the necessary
functionality.

(3) no one person, and probably no small set of people, know or understand all
the ways the system is used. Somehow, you have to capture that knowledge and
use it to create a functioning new system. At least the medical record company
I worked for hired lots of doctors and nurses out of hospitals to help them
understand how users used the system in practice. You essentially don't
understand at all how prosecutors, police, secretaries, and all the other
people involved in law enforcement use this system.

There are clear parallels to replacing systems like this. Particularly when
you have to do a live transition.

~~~
japhyr
I recognize the difficulties of attempting a project like this, and the
serious ramifications of any bugs you introduce into the project. There are
life and death ramifications here as well - contributing to a wrongful
imprisonment, or accidentally freeing a dangerous criminal are pretty bad
outcomes. But if no one attempts this, those things are going to happen
anyway.

I'm not saying I could do a perfect job - just that I imagine I'd have
something to show for working on this problem full time for 10 years, with the
stated level of funding.

~~~
x0x0
I don't doubt you would.

But the challenge in these projects is generally not technical, which your
post focused on. It's all management: management of scope, management of
requirements gathering, management of expectations, management of
deliverables. Management of having to work with people who, frankly, are dumb,
and for whatever reason resent you, but they use the system and you have to
figure out what they need anyway.

Oh, and the better question in all of this is why is Alaska building such a
system? Are they such special snowflakes that not a single other state or
country in the world has a similar system that they could buy and minimally to
moderately adapt for local needs? Forty-nine other states, plus DC, plus
puerto rico, plus the fbi's system, not to mention the provinces in Canada or
states in Mexico. Surely someone has a system that could serve as a 90%
starting point.

~~~
mlinksva
What is the best guess at answer to the better question?

Relatedly, why aren't these systems mandated to be open source? The public
ought be able to inspect the algorithms that regulate them, and various parts
of government ought be able to collaborate freely.

------
smoyer
One of these days I'd like to volunteer (with a large enough group of peers)
to build a system like this for the government free-of-charge. I can't imagine
that a hundred good developers working part-time couldn't get this system
migrated in a reasonable amount of time.

One clue to why if failed describes there being a tension between departments
and a new governance board. Providing a system like this based only on best
practices should be pretty feasible if you throw out all the petty politics
and "perceived but useless" pet features.

I'd imagine the current system (as it's mainframe-based) pretty much follows
the typical forms entry (perhaps they're still using green-screens?). Moving
the existing system as-is to a modern version of an RDBMS with an equivalent
web-based front-end should be the first step. Once you're current, you can add
features as you go.

~~~
japhyr
I'm not sure that a hundred good developers would complete a project like
this. I'd rather take a small team who could dig deeply into the project, and
commit full time to it. One or more of those people would have to be pretty
politically savvy to help pull it off.

~~~
acveilleux
It's death by a thousand tiny procedural cuts. There's thousands of users,
dozens of processes and as many external dependencies.

Money is always subject to whomever got elected last November and a great many
people resent you. They're 57, been working with the previous system since the
mid 80s and they like it dammit. They see the writing on the wall and the
mainframe operator or data entry specialist job they have is a goner. Shy of a
full pension too.

Or the prima Donna department that always got their way is blue balling the
analysts the whole way through because they don't own this anymore.

Problems in this type of projects are all people and processes. I also work in
Healthcare IT on the vendor side and I see projects that get shit done and
project that die. It's about management.

~~~
spiritplumber
Yep, I can confirm that.

My strategy is to tell people "I am going to make it work and you are going to
either help me, or watch helplessly as I make it work without you and make you
look like an idiot".

It helps to take assertions like "you cannot do that" literally, even though
they tend to be said to mean "you may not do that" or "you should not do
that", and prove that you can in fact, do that all you like because there are
no laws of physics preventing it.

It has caused a few surreal meetings, like the time when a guy did a 40 minute
long presentation on "I need more people and more money in my workgroup to
make this work" while the first batch of completed, working, tested production
units (built by me and one other guy after we got sick of the BS, I even paid
for the manufacturing out of pocket) was in the middle of the conference
table.

Imagine a half dozen people trying to ignore what was literally in front of
their eyes.

Another time it also has caused me to escort a security guard out of the
building because he was being disruptive. He had been told to escort me out of
the building, grabbed me without asking my permission, and found himself in a
headlock, then was informed that if he did want to fight me, he was welcome
to. Then I got back inside and finished my business.

Sometimes you just have to say "full speed ahead and damn the torpedos", or
nothing gets done.

(My father's rule of fights: Never hit first, always hit last)

~~~
spiritplumber
I love how most of my comments that convey the message "Sometimes you have to
push back when you are pushed" or "sometimes the job is more important than
people's egos" get downvoted.

------
nova22033
>The company is also known for big-ticket software development contracts with
federal government agencies, from the National Security Administration to the
Department of Defense to the FBI.

big ticket projects for the FBI? Like VCF?

[http://www.washingtonpost.com/wp-
dyn/content/article/2006/08...](http://www.washingtonpost.com/wp-
dyn/content/article/2006/08/17/AR2006081701485.html)

> the system delivered by SAIC was so incomplete and unusable that it left the
> FBI with little choice but to scuttle the effort altogether.

------
yeukhon
As much as I hate Oracle / IBM contracted software (my alma mater paid Oracle
more than $600 millions to replace our university portal, and half of what
they did was customize existing ecommerce system to fit our need), compare to
this project, I feel much better about the new portal Oracle delivered to my
alma mater.

Look, I am okay with the contractor subcontracts to 34 other contractors. I
will take that as hiring consultants from 34 other consulting agencies. But I
can bet with my life they were all working in vacuum and in a full water fall
model. No doubt we need to start with analysis, and propose a possible
solution like database schema and migration, but one can still work with other
parts of the project iteratively, e.g. front-end and some other micro-
services. I also suspect contractors hire offshore which can further
complicate the development process as one has to deal with language barrier
(kind of a minor issue) and timezone difference. FYI I do work with offshore
day-to-day and the one I work with are excellent.

Like healtcare.gov, and the fictional (but based on real life stories) Phoenix
Project, there was little to no human resource and IT capacity planning, and
management oversight is completely broken with 35 contractors involved.

I cannot underestimate the complexity involved in migrating old data, but I
bet if we gather a bunch of database professors they can probably come up with
a pretty good database migration plan and a new database schema design.

~~~
zaroth
Just like healthcare.gov, the reality is you start with an absolutely trivial
problem and turn it into a nightmare. CRUD is CRUD. The complexity is imposed
by project managers and engineers that can't help themselves or simply don't
know better. Seen it happen over and over, and often just the effort to _stop
the madness_ is more work than a fully working sane implementation.

More than once I've seen stealth mode side-projects run on a single engineer's
own time out-deliver and out-perform the fully funded project because the
standard approach is to take absolutely trivial problems and heap on absurd
layers of complexity which of course are full of bugs and utterly fail in the
field.

I wonder if taxpayers have ever tried to sue for damages, but I'm sure it's
impossible.

~~~
HillRat
_Just like healthcare.gov, the reality is you start with an absolutely trivial
problem and turn it into a nightmare._

I wouldn't call hc.gov "trivial" \-- it started with all the fun of a massive-
scale website, added X12 EDI workflow, then finished up with real-time
communications with high-security government systems never meant to be
interoperated with and at least a few of which probably can't communicate over
red-black networks. _Then_ they started adding complexity to it.

Admittedly, some of those added complexities caused major headache, such as
the requirement to use ORCL IDM (which, if the hotwashes are correct, proved a
constant Achilles' heel to the ops side -- who the hell put _that_ requirement
in the spec?), but the project started out high-risk.

The problem with government projects by the way, isn't with "project managers
and engineers;" it's with the dozens of agency stakeholders, half of whom are
political appointees and the other half of whom are waging bureaucratic turf
wars against everyone else. With high-profile projects like hc.gov, you also
get politicians involved, so everyone starts freaking out about things like
the fear that some illegal immigrant working in a Kansas poultry
slaughterhouse could accidentally qualify for health insurance and end up as
the lead story on FOX News. And, of course, even though the seventh floor
(and, in this case, the Oval Office!) gives lip service about how critical it
is that the tech works perfectly, you'll never see an appointee or an SES
employee actually try to keep the ship upright.

After that, add in large-scale government contractors that operate on a high-
risk version of waterfall that requires every single piece to come together
perfectly, on time, without errors, and for the internal specs and contracts
to not shift _at all_ \-- well, it's a recipe for a massive clusterfuck, every
single time. There's a reason Medicare is still running on mainframes and FERS
has never gotten off paper -- the risks of failure are just overwhelming.

~~~
zaroth
You sound really knowledgeable about hc.gov back-end implementation, and I
think everything you are saying has just proved my point. :-/

~~~
HillRat
I only know what was in the trade press! However, I've familiar with how both
government and enterprise projects in general evolve (or, to put it another
way, I've been kicked by that mule more than once).

------
jonas21
$28 million? That's nothing. The Air Force recently spent over $1 billion on a
cancelled logistics software project.

[http://www.nytimes.com/2012/12/09/technology/air-force-
stumb...](http://www.nytimes.com/2012/12/09/technology/air-force-stumbles-
over-software-modernization-project.html)

~~~
adwf
$1 billion? That's nothing. In the UK we spent over £10 billion on our
abandoned NHS IT system ;)

[http://www.theguardian.com/society/2013/sep/18/nhs-
records-s...](http://www.theguardian.com/society/2013/sep/18/nhs-records-
system-10bn)

~~~
arikrak
That's something, even more than the US government spent on the Obamacare
website.

------
imperialdrive
I'm just starting to flesh out a new database system for an SF government
related agency, ground level, replacing something almost 15 years old. If
there is a single person or two out there that wants to sit in on a meeting,
brainstorming, it would be much welcomed. 501c3 org. I'm not even a DB guy,
more of a general tech solutions architect, but I'm curious (as others on here
are) just how much waste is necessary with these kind of projects. So far
we've spent about 20k on maintaining the current system, maybe even more over
the past 2 years. Anyway, I could go on and on, but if you're in SF, drop a
line for lunch/coffee on me, dan at sfpretrial . org

------
earleybird
If the organization is dysfunctional the project is doomed from the outset.
From personal experience I've been in or seen more dysfuntion than function.
Probably the most illuminating quote from the article: "The failed project
exposed simmering layers of tension within the Department of Public Safety and
its division that supports law enforcement technology and criminal history
information statewide, documents show, and has triggered numerous
restructuring efforts, including a new internal governance board."

------
aaron695
IT Conversations had a good talk on large scale databases and why they fail.
($170 million FBI database)

[http://cdn.itconversations.com/ITC.IEEE-
FBISoftware-2006.06....](http://cdn.itconversations.com/ITC.IEEE-
FBISoftware-2006.06.01.mp3)

TL;DL No one actually sat down and specifically told the contractors what was
needed because they didn't bother to even work it out for themselves.

So the contractors could get away with what they wanted.

------
omouse
This is why software estimation and as much honesty as possible is required
when dealing with clients. For all the talk of risk management, clients
_still_ believe the most optimistic schedules! Whatever a sales person or
manager says, pad it by 50-100% and if you can live with that, then you can
ask for updated estimates every month to increase accuracy.

------
TYPE_FASTER
Here's another large contract failure:

[http://www.nytimes.com/2013/11/23/nyregion/3-found-guilty-
in...](http://www.nytimes.com/2013/11/23/nyregion/3-found-guilty-in-citytime-
corruption-trial.html?_r=0)

Multiple large organizations on a large scale consulting engagement seems to
lower the chances of success.

------
AndrewKemendo
There are literally thousands of projects like this throughout the US federal
government.

------
coldcode
"In an email, a spokeswoman for Leidos would only say that “the Leidos system
was delivered on time, on-scope and is fully functioning.”" I can't imagine
saying that with a straight face.

~~~
jobu
Yeah, given the end result I can't imagine what they're thinking by saying
that. To their credit, the auditors said this was their first negative review
of the company:

 _" Charles Collins Jr., the lead author of the MTG report, said in an
interview that the consulting group had encountered SAIC in the past. He said
he believed this to be MTG’s first critical report on the contractor."_

Their review of another department (Division of Statewide Services) was
particularly damning though:

 _" In the December 2013 “organizational review” of the Division of Statewide
Services, also conducted by MTG Consulting Group, auditors found evidence of
what they called “a seriously dysfunctional IT support organization.” The
review found that the division lacked governance and communication, its
benefits to other divisions and departments were unclear, and there was a
serious aversion to risk or change."_

Sounds like the place was being run by Mordac:
[http://goo.gl/o7JRvr](http://goo.gl/o7JRvr)

------
falcolas
> The replacement system was to be built from scratch

The downfall of many a project. Most just don't accumulate $28m in costs
before they fold.

~~~
x0x0
Given the ratio of expense to scale of the system, I doubt that's true. Many
similarly scaled projects fail inside private companies, but we just don't
hear about them.

------
fspacef
Bureaucracy at it's best

