
Why US government IT fails so hard, so often - Jtsummers
http://arstechnica.com/information-technology/2013/10/why-us-government-it-fails-so-hard-so-often/
======
DanielBMarkham
I just got through speaking in DC about Agile, startups, and functional
programming, mostly to an audience that is intimately familiar with government
work. It's really bad when you try to be diplomatic about how bad things are
fucked up and get open laughter from the crowd. There's no BSing people who
live this everyday.

Talking with folks afterwards, I heard about systems that I could have
rewritten in the late 90s in a few months that are still in the process of
being rewritten, 15 years later. After literally hundreds of people have
worked on them costing the taxpayer millions of dollars. And these are not
unique or oddball edge cases. Stuff like that is standard operating procedure.
I can't impress on the HN audience enough how terrible federal IT is overall.
(Aside from some really great outliers). There's a vast and increasingly large
gap between political rhetoric and the reality on the ground. This has been
true for at least the last 20 years.

I take issue with some of the details the supporting argument makes here, but
that's mostly a quibble. The author is definitely headed in the right
direction.

I'm a veteran of several federal IT programs, and a real veteran. I have seen
some really good federal teams, but never a good federal team of teams. It all
falls apart once you get past a few people and start throwing some kind of
structure and management in place. I understand that Agile is becoming
required for some federal work, but I'll bet you a thousand bucks this kind of
"Agile" isn't going to be the kind that the rest of us use. Think big tools,
big management, lots of stupid measurements, and no accountability. All with
the word "Agile" stamped on the front of it.

 _...The bottom line is that federal IT programs ' success is measured by
things that have nothing to do with how successful they are or by the metrics
most of the world uses. While the business world (and Web companies in
particular) now monitor user experience and productivity as a metric for IT
success, the government keeps throwing out numbers that mask the truth: the
only people who would use their systems are the ones that are forced to..._

Yep.

~~~
Jtsummers
"Agile" when federalized will probably turn out like "Lean". Lean in
manufacturing and similar processes, when done right, is a good thing.
However, the government version of it is almost a complete perversion of the
intent. Instead of bottom up, employee-driven changes, it's top down based
strictly on metrics and the changes typically effected so that whoever's
currently in charge can say, "I made <process> 10% more efficient by reducing
the number of wasted <things>." Employees resist the changed
processes/taskings because they fear a loss of employment, and are kept out of
the loop when the changes are being devised. Federalized Agile would likely be
similar.

~~~
DanielBMarkham
Yep.

We saw this same thing with source control. Who doesn't like source control?
But the government, in an effort to make sure there was absolutely no risk,
mistakes, or accountability associated with software development, started
adding more and more mandatory forms that had to be completed each time a
developer checked something back in.

This created such a mess for developers that it wasn't unusual for one check
in to take hours to accomplish. You'd read a book while the system tried to
accept your code changes.

Agile is all about failing fast and learning quickly. Most contractual work
for the government does not include the possibility of failure. At all. In
fact, the way the contracts are written is such that the contractor takes on
all responsibility for everything, even things there's no way they could
control. In response to this idiotic situation, contractors start bidding at
crazy low rates on these things, say 40 bucks an hour, then actually make
their money on changes. So it's always in the contractors interest to capture
and execute changes, while it's the government is trying to get the world
delivered to them on a platter for five bucks. People use the term "perverse
incentives", but this is like that taken to the next level. It's a multi-
layered thing. It's total systemic failure. We'd call it systemic collapse,
but the money keeps flowing in anyway, so that term isn't applicable, although
it certainly feels like it.

This kind of distrust doesn't make for anything but a living hell for
everybody, government employees and contractors alike. So, with tons of rules
and structures in place, the way things really work is through relationships.
People who are easy to get along with and helpful get more government work.
Both sides of the fence can't control the process, and nobody rocks the boat
and everybody gets a piece of the pie. You end up from the government side
with some sort of informal and unspoken version of _I don 't care if you make
a lot of money, just make sure me and my friends look good_ Also it'd be nice
to pick up a little lucrative consulting work once I've put in my 20 and can
start double-dipping.

Your average developer has absolutely no idea about this. If they've worked in
IT in federal work, they probably have horror stories, but the more you see
the worse it gets. Oddly enough, in the best cases you end up with people who
really like working together, are motivated, but spend most of their time
fighting a Sisyphean fight against the idiocy of the system.

~~~
mtts
> But the government, in an effort to make sure there was absolutely no risk,
> mistakes, or accountability associated with software development, started
> adding more and more mandatory forms that had to be completed each time a
> developer checked something back in.

That's because "the government" is operating under different pressures than
corporations. A corporation, big or small, screws up, only a handful of
stakeholders are affected and raise a stink. If the government screws up,
you're looking at headlines in the Washington Post.

Plausible deniability is a powerful force.

~~~
Jtsummers
A nit: This isn't plausible deniability, but rather CYA (cover your ass).

Plausible Deniability: Delegation of tasks and lack of records of
communications letting the actual responsible party deny knowledge of the
"insubordinate" or "rogue" underlings.

CYA: A process to change the process to define the process for tracking
changes to the process, with 20 signatures or more required at each step of
the process, digital forms filled out in triplicate. Or, making proper
recordings of communications with underlings so when they screw up you can
point out that they didn't act according to your direction, and that they
failed to report issues back up the chain.

------
GVIrish
This article is only touching on a small portion of why government IT is
terrible.

People like to blame contractors but I think a large part of the problem is
that the federal employees who are responsible for these big projects are
rarely held accountable for failure. In fact, the lack of accountability for
federal employees is why so much work was farmed out to contractors in the
first place.

Then there's the fact that a lot of the decisions in government IT are made by
people who have little understanding of how successful software gets built.
People don't understand the importance of defining the scope of the project
properly. Or you get people that change requirements willy nilly. Or you find
people that are just trying to protect their little fiefdom so you end up with
wasteful duplication of effort or withholding of vital information.

The Ars article indirectly touches on the fiefdom issue. When you have
dysfunctional IT departments or applications, often times instead of fixing
the dysfunction, someone just creates their own little separate IT effort to
get around it. So the VIP's might get something functional while the peons get
classic ASP & Access.

What I find very interesting about the Healthcare.gov problem is that this may
be the first time that a major federal IT project failure has major political
consequences for elected officials. If Healthcare.gov ends up being a crash
and burn failure, it could result in Republicans gaining enough power in the
next election to overturn the whole ACA.

~~~
RougeFemme
I don't think lack of accountability is what pushed the work out to
contractors. And if that is the case, does that mean the accountability
"stick" got pushed out to the contractors. I think work got pushed out to
contractors from a desire to out-source and because contractors, for various
reasons, generally do a much better job of staying abreast of best practices
than the government employees. And I say this as someone who has worked on
both sides of the fence.

I agree that requirements are a _big_ issue. In addition to the requirements
issues you mention, there is the problem with the "business line owners"
attempting to unnecessarily dictate implementation details, poorly defining
the functional requirements in the first place and ignoring interfacing
systems.

~~~
GVIrish
Where lack of accountability comes is the idea that contractors can be fired
if they don't do a good job, whereas it is much more difficult to fire a full-
time employee. So rather than solving that problem, most of the government's
software development is farmed out to contractors. The other reasoning for
contractors is that contractors don't get retirement benefits that FTE's do,
so in theory that saves the govt money but in practice it probably doesn't,
given that contractor bill rates are usually 200-300% or in excess of what FTE
salaries are.

Not to say that FTE's are great or anything, contractors do tend to me more
up-to-date on technology but I think that is a result of the government
employee hiring practices and compensation scale.

Yeah I've seen a lot of "business line owner" issues like the ones you've
mentioned. Another common one is that software is often built to mirror
dysfunctional and wasteful business processes. So you either get software that
is needlessly complex, or you build the software then have to re-do it because
people realize how bad the underlying business process was.

~~~
acdha
> Where lack of accountability comes is the idea that contractors can be fired
> if they don't do a good job, whereas it is much more difficult to fire a
> full-time employee.

When was the last time you saw that happen? The only major .gov IT screwup
which I can recall having any consequences for the vendor was the FBI's
virtual Case File system. Everything else tends to get brushed under the rug
either to avoid disrupting the relationship (!) or because nobody wants to
have to explain why they didn't say anything before the failure was publicly
obvious.

~~~
GVIrish
Exactly. The idea is that contractors can be fired for a poor job, but the
reality is that it rarely happens. Federal employees are rarely held
accountable for their failures so in turn they rarely hold contractors
accountable. Maybe that contractor doesn't get to stay on for option years,
but many times they do for a myriad of reasons.

Then there's the incestual revolving door between government and contracting
companies. Government lead retires, then goes to work for the contractor they
used to oversee. The biggest contractors use that as a prime mechanism for
earning new business. In some companies the best path to get into senior
management positions is by being a former Fed.

~~~
acdha
I don't think increasing accountability would solve every problem in
government IT but it's hard to under-estimate how many problems it would
solve.

------
specialist
I've done IT at both megacorps and govt. Frankly, I don't see a difference. If
anything, govt work has greater transparency and accountability.

Contributing factors are, but not limited to:

\- scale

\- complexity

\- legacy / installed base

\- laws, regulations, rules

\- organizational processes (workflows)

\- quality of developers

\- resources

\- politics and agendas

Just trying to support all the entrenched chaotic mutant use cases and edge
cases will wreck any project plan.

I much prefer working at younger, smaller companies. There's fewer roadblocks.
But the healthcare and other benefits working for government rock, so here I
am.

------
kilroy123
Isn't this how any government project works? IT, construction, etc. Sees like
the government is terrible at building anything, especially when contractors
are used to do the job.

Probably why the government is so big and expense. I can only imagine what
that $600+ billion gets us for the military. -_-

[http://en.wikipedia.org/wiki/Military_budget_of_the_United_S...](http://en.wikipedia.org/wiki/Military_budget_of_the_United_States)

~~~
iskander
> Sees like the government is terrible at building anything, especially when
> contractors are used to do the job.

Strange to contrast that with a tremendously active and efficient era of
public works from the 1890s through the 1950s. Just thinking of New York City,
I suspect that the total cost of all the bridges, tunnels and highways
(adjusted for inflation) might add up to a lot less than the 800 billion USD
that disappeared into our recent stimulus bill. What changed?

edit: Apparently the Brooklyn Bridge cost ~15 million USD to originally
construct, which now would be somewhere between 300 million to 400 million USD
([http://historical.whatitcosts.com/facts-brooklyn-
bridge.htm](http://historical.whatitcosts.com/facts-brooklyn-bridge.htm)).
Even though these are rough figures and inflation over large time horizons
isn't a perfect measure, I'm still astounded. The government's totally broken
healthcare website cost more to build than the Brooklyn Bridge.

~~~
Jtsummers
The relationship between government and business, the much more short term
approach to making money for businesses and individuals, a loss of the concept
of "public good" or "common good".

~~~
jerf
I'm sorry, I don't really see how any of those explain the government being
granted money (in _staggering_ quantities), then spending it, then having
nothing. Are you implying that because businesses have a short-term approach
to making money that therefore, the government is forced to give money to
unproductive contractors? Are you saying that because "the people" don't have
a concept of "public good" that when the government bids something out they
can't help but use a bidding approach that virtually guarantees failure?

Vague platitudes about society at large aren't crashing government projects.
Society at large aren't really involved in them, after all. Something specific
between the interaction of the government and the government-chosen
contractors is probably the problem.

In a way, you're probably proving far more than you mean, accidentally; if the
problem really _is_ "society at large", then there is no solution and these
large projects can't work, and therefore the most effective thing the
government could possibly do is _stop trying_ and wasting money. (A project
with a prerequisite of "change the larger culture's attitudes" is simply a
guaranteed failure, after all.) I suspect that's not the point you're going
for. You probably _really_ want to be arguing for a much more local reason.

~~~
joering2
> I'm sorry, I don't really see how any of those explain the government being
> granted money (in staggering quantities), then spending it, then having
> nothing.

Would it help to know that the same government (altho Federal Reserve is
private but heavily regulated) print the money for their own use?

Imagine you have a small house with a garage inside and printing money machine
in it. Now you want to expand the house and you know you can pay for the
materials labor, etc, thanks to your printing machine. Wouldn't you do it?

The answer is those who care about US, about their outlook on the world would
be very careful with using this machine. Those who would "own the house" for
just a little bit would abuse it. We are at this stage where government human
resources are so overbloated, ,you need so many signatures to buy a roll of
toilet paper that eventually at the end of the day nobody is responsible for
anything.

That can least if you own the world and don't have to answer to anyone.
Un/fortunately this is not US case. If we default, we lose credibility with
other major player such as China. Imagine if China stops producing for US
(would be insane embargo if you think about out), can you imagine what would
happen with us economy? all of sudden the soap dispencer you buy at wallmart
cost $45 (made in USA) instead of 4.99 (made in China). Unfortunately, like I
said there is no credibility even Chief in Charge Barrack Husain Obama is not
aware (in most part) what is really going on under his watch.

~~~
mikeyouse
Wow that was a load of nonsense. You don't actually know what the Fed does, do
you?

And;

    
    
        Chief in Charge Barrack Husain Obama
    

Your awkward attempt at slurring the president actually involved spelling two
of his names wrong.

------
cjoh
This is all right, but a little vague. Here are some more specific problems:
[http://blog.dobt.co/post/63655420372/how-healthcare-gov-
went...](http://blog.dobt.co/post/63655420372/how-healthcare-gov-went-wrong)

~~~
GVIrish
This is an excellent analysis. Particularly the point about setting the go-
live date long before the scope of the project was understood, not to mention
the fact that there's no way they could have had an open bidding process AND
meet the deadline. In some ways it's a miracle this went live on-time, flaws
and all.

However, I suspect the true extent of the flaws with Healthcare.gov and the
exchanges is not understood yet.

If anything good comes out of this, I hope it's that the government finally
starts paying attention to the woeful state of government IT projects.

------
tjaerv
Surprised they didn't mention the all-time fiasco of the FBI's disastrous
$170M Virtual Case File project:
[http://en.wikipedia.org/wiki/Virtual_Case_File](http://en.wikipedia.org/wiki/Virtual_Case_File)

~~~
DanielBMarkham
That's just small potatoes. The really big stuff is almost invisible. That's
because the liability is too large, but every now and then you can see how
things are going. Check out the IRS modernization efforts. (The quote and link
are probably 5-10 years old. Haven't heard much about it one way or the other
lately)

 _...The IRS has failed twice to modernize its system, in the 1970’s when
taxpayer privacy issues surfaced and in the mid-1990’s when the IRS spent ten
years and $2 billion on the project and had little to show for it. The current
project launched in 1999 with a budget of $8 billion. The project is late and
over-budget; problems have been well publicized after the release of the IRS
Oversight Board investigation..._

[http://www.kellogg.northwestern.edu/student/courses/tech914/...](http://www.kellogg.northwestern.edu/student/courses/tech914/summer2004/projectfailures/projectfailures/irs2.htm)

AFAIK, the IRS dropped $10 Billion in IT money in just over a decade with
little to nothing to show for it.

I used to joke around and say heck, they could have paid me 2 Billion and I
would have also provided them nothing, but look at the money they would have
saved!

~~~
tjaerv
Thanks for that, I had no idea.

~~~
DanielBMarkham
The general rule is: the larger, older, and more politicized the agency is,
the more awful the technology development.

So NSA, which is fairly young, stays out of the political eye (until
recently), and small? They do some really cool stuff. But the older, more
controversial agencies, covering things like immigration, defense, taxes,
health, and so on? Not good.

------
mmagin
At an individual level: How many programmers who are good enough to get a job
that pays well in the private sector want to take a job (either in the
government or for its contractor) with all that bureaucratic overhead?

~~~
vonmoltke
More than I think most people here realize. Some people find particular,
unique problems fascinating. Others have a sense of duty that outweighs
material or personal concerns. Others just don't know better.

------
uptown
Yet if you look at what's been revealed about the NSA's capabilities, they're
excelling in some areas of technological execution.

------
djhworld
Here in the UK most government contracts just go to the big enterprise shops
(think Accenture, IBM, Logica, Cap Gemini, ATOS etc) who manage to negotiate
fairly hefty long term deals and charge a healthy commission for taking them
on and running them for years.

This is why you run into legacy codebases that are 10-20 years old and will
keep on going.

------
eitally
The reason it fails is because of articles like this disincentivizing the
talent they need from being interested in working for either a) .gov/.mil or
b) government contractors. Additionally, the contracting process being
divorced from the project execution weighs heavily.

~~~
Jtsummers
Articles like this don't disincentivize a lot of people, and burying it would
just let them be lured in and find out the hard way how bad it is. Articles
like this air a problem that needs to be corrected.

What disincentivizes talented (specifically young talent) people is the ageism
inherent in the system and its perverse method of rewarding people
(specifically talking civil service here, contracting's another beast). Unlike
the ageism you hear about in the occasional article about Silicon Valley, its
the young that have a hard time in civil service. Positions aren't available
because too many people are waiting to retire (for various reasons). When they
do get in there are few leadership roles open for advancement (see previous
sentence). Their pay is strictly time and not merit based (well, not 100%, but
NSPS is gone and only a handful of places use its equivalent of merit based
pay increase). Career fields are disincentivized, for instance engineers are
treated better than computer scientists and mathematicians even in offices
where the latter make more sense (bonuses, faster promotion schedules for
trainees, pay differential for the degree). Lack of proper training programs
(this is office by office, so not sure how universal the problem is, but it's
pretty shabby in a lot of places). People are expected to be hired in and,
within months, become experts on arcane and outdated systems, with no support
because the actual expert has just retired (that's whey there was an opening).
People leave quickly, similar to the Teach for America article earlier in the
day, because they burn out or realize there's no where to go, neither up nor
down, just steady.

On the other hand, in some regions older, talented folks can get a good deal.
They can come in as a high grade GS employee with a middling step,
automatically enrolled in a decent 401k (5% matching), good insurance (less
critical with Obamacare?), and, after a couple years, effective tenure so they
have stability which they might value for their family.

~~~
richieb
Not much stability with the current Congress.

------
EpicEng
A new version of Windows Server doesn't address the problem of loading 64 JE
files and 11 CSS files on every page. There is some programming ineptitude at
play as well, though I realize that the article was focusing specifically on
the new ObamaCare site.

~~~
acdha
That's also a minor side issue: it'll cost you a sub-1-second page load time
but it doesn't cause the kinds of problems people are talking about where the
backend components are overloaded to the point of failure.

The size of healthcare.gov is actually well under the average:

[http://www.webpagetest.org/result/131010_ZG_X5C/1/details/](http://www.webpagetest.org/result/131010_ZG_X5C/1/details/)

(see [http://www.webperformancetoday.com/2013/06/05/web-page-
growt...](http://www.webperformancetoday.com/2013/06/05/web-page-
growth-2010-2013/))

For comparison, look at WashingtonPost.com: it uses considerably more data and
more total requests and yet the page loads much faster:

[http://www.webpagetest.org/result/131010_GG_XDK/](http://www.webpagetest.org/result/131010_GG_XDK/)

If you look carefully, though, you'll see a) that they aren't setting Cache-
Control headers and are setting ETags (both performance killers) and b)
there's this weird staggered response behaviour which indicates something is
seriously wrong. It seems to handle much better in Chrome - 3s vs 70! - so
presumably there's some interaction with the browser going on as well:

[http://www.webpagetest.org/result/131010_ZW_XD6/1/details/](http://www.webpagetest.org/result/131010_ZW_XD6/1/details/)

------
ihsw
Large organizations need homogeneous computer systems, so it stands to reason
that centralized configuration management would be a _perfect_ fit.

So why hasn't the likes of puppet and chef caught on? Genuinely curious.

~~~
jl6
Large orgs want guarantees and certainty, and so typically look for software
support from other large organizations. Easier to hold IBM to account than a
small independent vendor who may have gone out of business by the time you
need them. Yes, this is an expensive approach.

Who is selling support for Puppet and Chef?

