
The way government does tech is outdated and risky - madelfio
http://www.washingtonpost.com/blogs/wonkblog/wp/2013/10/21/the-way-government-does-tech-is-outdated-and-risky/
======
squidfood
Federal Gov employee here. Can't speak for a project with this scope, but the
procurement middlemen get into everything, far for the worse.

Two years ago our team wanted to buy a small cluster (~300 cores, ~$50K). We
talked directly to two good vendors (good recommendations from university
partners) and came up with a fine machine and 2 bids for it. Sent
recommendations to procurement.

Procurement put it out for bid, and a fly-by-night company undercut the bid by
$10K... by noticing that procurement had not specified details of service
level (that were in the bids we'd gotten and forwarded). Procurement, once it
goes there, is a true black box. No communication, no understanding.

Five months later, we were basically delivered 2 pallets of unassembled parts
and no instructions. Believe me, we spent 3-4x as much in labor as the $10K
savings to get it working, and it's been plagued with issues that would have
been under the onsite service warranties for the better companies.

The biggest irony is: I firmly believe that procurement acts this way not
because the government is fundamentally incompetent, but because the Public,
and thus Congress, BELIEVES we are incompetent, so puts so many levels of
"check" bureaucracy in place that the people who know what they want can't
participate directly in getting it.

~~~
joshdick
Procurement rules are more to guard against corruption and cronyism than
incompetence.

Without them, you could reward your friends with fat government contracts,
regardless of what's in the public's interest.

~~~
apendleton
The irony is that they don't really do that. It's common in both procurement
and hiring in the federal government to see bizarrely-specific requirements
whose fairly transparent intention is to limit the potential bidder or
applicant pool to a single company or individual.

~~~
brianpgordon
* Must have a company name which SHA-1 hashes to e8b06511aa36381fc2306eb6f8181204585c5453.

Hey, there's a theoretically infinite number of company names which meet this
requirement...

------
crbnw00ts
That diagram for the "waterfall" approach that they yanked from Wikipedia is a
complete straw-man representation. It's nonsense.

Here is the actual, original source for the Waterfall approach, first
published in 1970:

[http://leadinganswers.typepad.com/leading_answers/files/orig...](http://leadinganswers.typepad.com/leading_answers/files/original_waterfall_paper_winston_royce.pdf)

If people would just bother to scroll past the first couple of pages, they
will notice that the approach _already includes_ some iteration cycles between
steps.

In other words, this whole "agile vs. waterfall" debate which has wasted
countless hours of human effort is based on a _complete misunderstanding_ of
what "waterfall" is in the first place. No one ever seriously proposed a model
without iteration. It simply never existed in the first place!

~~~
001sky
_" What often happens when you have these big requirements up front, is the
people who are specifying the product are afraid of not getting all their
ideas in, so they overscope the project. And then the development team is on
the hook for delivering everything, not just the essential elements."_

Isn't this the essence of the critique, though? Its not the lack of iteration,
but the logic of the spec-formation.

To put this another way, you are right that it has nothing to do with the
waterfall model per se. But it has everything to do with accepting the same
set of behavioural assumptions. Namely, that the people who spec the model
have perfect foresight regarding the spec itself.

What needs to be accepted is "incomplete spec". The problem with this is then
the hold-up problem. you get held to ransom to fill in the incompletions. So
what is really needed is a capability to execute this more in-house. This
would prevent the re-negotiation of the economics (the hold up), because the
project manager would just execute properly, directing resources (already paid
for) through fiat rather by then re-negotiating vs a modified spec.

One problem with this is accountability. There would need to be more
accountability, because execution of the incomplete spec will not be the same
as farming responsibility for the spec out to a third party vendor.

 _Project managers need to get used to the idea of working intensely with a
development team rather than asking for a specific thing and then walking
away_

~~~
Swannie
Here here!

It would be lovely if vendors would bid based on their experience, and be
compensated for it, at a time and materials basis. The more experience and
better you prove to be, the higher we're willing to pay for a better outcome,
sooner.

Except that world is rife with bait and switch. And writing the code that
delivers the spec is just a small part of the picture. Companies are terrible
at specifying the non-functionals, delivery process expectations, operational
requirements, supportability requirements, etc. When they do get it right, the
costs go up, because many places quote without any concept of these things.

The sad reality is that the people who get asked to quote for these things in
the software world are generally clueless about the actual domain, out of
touch, and wildly wrong most of the time. And the people that specify these
things are often barely any better. And lets face it, software developers are
also terrible at quoting times to do a task, though that can be mitigated with
a lot of experience of that task and the code base to do it on.

------
leelin
For the MIT alums out there, I remember a 6.170 exam that had the question:
"When is it appropriate to use the waterfall model of development?"

The answer was any time you are developing software for the government! The
professor specifically mentioned it in lecture once, so that alone was enough
for full credit on the question (other reasonable answers were fine too).

Later I TA'ed the class twice and made sure to eliminate these pure lecture-
attendance-check questions.

~~~
mbesto
Interesting fact - the original design of "waterfall" isn't what we perceive
as "waterfall" today:
[http://leadinganswers.typepad.com/leading_answers/files/orig...](http://leadinganswers.typepad.com/leading_answers/files/original_waterfall_paper_winston_royce.pdf)

My theory - agile/iterative development rarely gets sold because we continue
to believe we aren't susceptible to planning fallacy.

~~~
svachalek
What do you perceive as waterfall? That's exactly the way we learned it in
school...

~~~
bpyne
Much of our industry believes Waterfall means a single-pass development
process. Dr. Royce explicitly said to do it twice. (It was a military officer
who later took the process and made it into a single pass.)

If you learned it as a two-pass process, then you learned it correctly.

~~~
zb
This is simply historically incorrect. Waterfall means a single pass by
definition.

Royce described the pre-existing state of the art - the single-pass model
(Waterfall) - and suggested a modification to a 2-pass model. (This can be
seen as a precursor to Boehm's n-pass Spiral model.)

To suggest that the single-pass model was invented later as a corruption of
Royce's paper is nonsense. Virtually all software was developed this way both
before and after the paper.

What is odd is that the earliest and most commonly cited reference to the
Waterfall methodology is a paper that explicitly says that it doesn't work.
Let this be a lesson on not burying the lede.

~~~
bpyne
In the version of the paper linked earlier in the forum, I take the following
to mean two passes: build a model and then use lessons learned to build the
final product. But it's a matter of interpretation and semantics. My own
interpretation is contradicted further down in my comment by someone from that
era (see the Larman paper linked). C'est la vie, I'm sticking with my
interpretation. I think we can agree that Royce did not mean our modern
"Agile" approach.

"If the computer program in question is being developed for the first time,
arrange matters so that the version finally delivered to the customer for
operational deployment is actually the second version insofar as critical
design/operations areas are concerned."

At least 2 methodologies existed before Royce's paper as described in this
paper: Waterfall and "iterative and
incremental".([http://www.craiglarman.com/wiki/downloads/misc/history-of-
it...](http://www.craiglarman.com/wiki/downloads/misc/history-of-iterative-
larman-and-basili-ieee-computer.pdf))

Note that, in the section referencing DoD-Std-2167, the author of the DoD
standard does state explicitly that he understood Waterfall to be one-pass.
Certainly he implicitly promoted it as such.

~~~
zb
The mistake I believe you're making is in assuming that because most people
reference Royce's paper for a description of Waterfall, that means that the
definition of Waterfall is whatever the main subject of that paper is. This
does not follow.

The paper describes a number (more than 2) of possible models. The main
subject of the paper is probably the "2-pass waterfall" diagram in Figure 7.
However, when people refer to this paper in the context of the Waterfall
model, they mean Figure 2. The Waterfall model is called the Waterfall model
because Figure 2 looks like a waterfall, with the water never flowing uphill.

Figure 7, however, was largely ignored. (Though Brooks later recapitulated it
as "Build one to throw away; you will, anyhow". I don't know whether it was
influential in the development of the Spiral model or not, though it seems
like a logical progression.) As far as I know this model never even got a name
attached to it. Its influence pales in comparison to Figure 2, which was the
first and best published reference to a design process that almost everybody
accepted as "ideal" at the time.

Note that when Brooks tells the story in _The Mythical Man-Month_ about making
the mistake of getting a large group of mediocre engineers to write
specifications instead of giving the job to a small group of elite engineers
because he didn't want the larger group just sitting on their thumbs for a
year, this was all happening in the mid 60s. Here's another contemporary
quote:

 _The most deadly thing in software is the concept, which almost universally
seems to be followed, that you are going to specify what you are going to do,
and then do it. And that is where most of our troubles come from. The projects
that are called successful, have met their specifications. But those
specifications were based upon the designers’ ignorance before they started
the job._ —Doug Ross, 1968

Winston Royce did _not_ invent the Waterfall model in 1970, he was just
describing the already-dominant paradigm as a prelude to proposing something
different. The model had been around forever, Royce provided a convenient
diagram (Figure 2!) in the course of trying to critique (or even replace) it,
and the name "Waterfall" got attached later.

> _Note that, in the section referencing DoD-Std-2167, the author of the DoD
> standard does state explicitly that he understood Waterfall to be one-pass._

Of course he did, and he understood it correctly. The thing he didn't
understand was that it was a _terrible_ idea.

Had he (properly) read Royce's paper, he would have seen that it actually
advocated a slightly better (but still pretty terrible) idea, but that idea
was not and is not the Waterfall model. The Waterfall model is the one in
Figure 2 that looks like a waterfall. Hence the name.

> _My own interpretation is contradicted further down in my comment by someone
> from that era (see the Larman paper linked). C 'est la vie, I'm sticking
> with my interpretation._

Um, OK then.

Thanks for the link though, it was really interesting.

~~~
hga
Note: in the 20th anniversary 2nd edition of _The Mythical Man Month_
([http://www.amazon.com/The-Mythical-Man-Month-Engineering-
Ann...](http://www.amazon.com/The-Mythical-Man-Month-Engineering-
Anniversary/dp/0201835959/)), in one of the new chapters, Brook's backs off of
his original "Build one to throw away" advice.

------
joeblau
This isn't going to change until there is a thought process shift for leaders
on both the government and contractor side. I worked on a project that fought
tooth and nail to create a project using an agile development process and it
was one of the best projects I worked on for the government. It was killed due
to politics, but the feedback, functionality, UX, and collaboration up to that
point were great. Everything else I did was waterfall and we always has the
same cycle.

    
    
      while (true) {
        // Contractor working for a year
        Government: This isn't what we wanted!
        Contractor: We met all of the requirements... See all of the boxes are checked.
        Government: Well we want to change 1,2...n things.
        Contractor: Okay, Let's do a follow on contract.
        Government: Okay, Here is the money; Go.
      }

~~~
nyrina
Sounds a bit like the contract I sent to India a few months ago.

One of my requirements was that it was able to send text messages.

What I got, was a ticked box, and an application that could also send text
messages to 1 number. No, a number I could choose. One number as in, one cell
phone.

And it wasn't even my own cell phone

~~~
joeblau
On their end they are probably saying "Mission Accomplished." I'm laughing
because of the situation, but that is just terrible.

------
joe_the_user
Has anyone noted that the government perhaps just shouldn't be in this kind of
business to begin with?

Any market based solution website has to be very agile and responsive [edit:
to succeed at it's goal] but the government can't be super responsive and in
many ways _shouldn 't_ be super responsive. The state spends all of our money
and enforces mandatory decisions concerning our lives. The state _shouldn 't_
have the agile qualities needed to produce the beautifully _flexible_ websites
created by the private sector.

In general, I'd claim the state should certainly be smaller but that it
shouldn't be less bureaucratic, shouldn't be more like a corporation. Civil
service is boring and bureaucratic _by design_ , specifically it was created
to combat the "spoils system" that plagued the early American state [1]
(though the prize of modern state eclipse what Tammany Hall etc could have
imagined). Modern corporations are agile by having a command structure which
lets them quickly maximize profits - which is great if we believe the market
system benefits everyone when operating properly. But states with the ability
to trample the fences of ordinary market shouldn't not be also given the
ability to move quickly and agilely to do this. Corporations have no internal
limits to their "greed" but we citizens of democratic market capitalism are
assuming that's OK, indeed desirable, as long as the corporation face the
strong _external_ limit of markets and individual choice.

The current fashion for what could be called "state-enforced private
consumption" is sold as giving us the best of all possible worlds but in
reality gives us the worst (IE. the reality is the wealth of this society is
indeed being vacuum-out by a kind of private-public rent seeker limited by
neither the traditional bounds of the democratic state or traditional bounds
of the market).

Note: I'm not a conservative rooting against Obamacare. It seems like it was a
terrible approach for achieving affordable healthcare but I still would prefer
it succeeded that failed because, well, I and many friends need it.

[1]
[http://en.wikipedia.org/wiki/Spoils_system](http://en.wikipedia.org/wiki/Spoils_system)

~~~
njharman
> Any market based solution website has to very agile and responsive

This is provably false by visiting any number of old, large company websites,
esp in healthcare or banking. In addition the government isn't in this kind of
business, it outsources almost everything to "free market" companies, who
extract as much margin as they and their lobbyists can get away with.

Complexity has much more to do with size of and number of evolved entities
than whether they are profit motivated or not.

~~~
john_b
Large companies in well established industries sometimes operate with near
monopoly power and are only threatened when the entire industry undergoes a
long term, permanent change. Most people don't consider such a situation a
functioning, healthy market. Healthcare and banking are two great examples of
industries whose dynamics tend to result in a small number of powerful
entities calling the shots, the former due to legal reasons and the latter due
to economies of scale.

~~~
Spooky23
Bad software development processes are everywhere. I've seen more than one
successful companies that aren't massive corporations futz around and produce
garbage. I've also seen highly effective government organizations produce some
awesome product.

~~~
hga
While I wouldn't call it "awesome" (bit too clunky for that), the Medicare.gov
site that CGI Federal is responsible for (don't know if they built it or the
recent Plan D shopping and enrollment part) is pretty good, very solid and
gets the job done.

------
buro9
The US (and rest of world) should take a leaf out of the UK's recent
initiative: GDS (Government Digital Services)

[http://digital.cabinetoffice.gov.uk/](http://digital.cabinetoffice.gov.uk/)

Aside from creating [https://www.gov.uk/](https://www.gov.uk/) which laid down
a lot of principles on how to fulfil a government contract (as well as the
foundations of what goverment websites should look like and how they should be
developed), GDS is also looking at the problem of procurement.

The GDS team essentially are wrestling back from the big contractors the major
contracts, breaking the work down into a large number of bitesize contracts
and then farming them out to a wide variety of smaller vendors.

So instead of finding a Fujitsu/Siemens JV team, or an IBM Professional
Services team, operating a £50m project, the plan is to offer 100 x £250k
projects to a large number of smaller suppliers instead. Each project having a
clearer purpose that is more able to be fulfilled.

Of course there are obvious overheads in managing so many projects, and of
course some of these projects will fail. But... overall the savings will be
such that the overheads are cheap, and the failed projects will only have a
smaller impact on a major programme initiative than a failure would today.

~~~
wozniacki
_This week a UK parliamentary watchdog described a failed National Health
Service patient IT programme – the cost of which has spiralled to £9.8bn – as
“one of the worst and most expensive contracting fiascos in the history of the
public sector”. Earlier this month the Department for Work and Pensions
admitted that it had written off £34m of IT costs, incurred in an attempt to
overhaul how social security benefits are paid. A week earlier Co-operative
Bank said it had written off the £148m cost of a new IT system that would no
longer be implemented._

[http://www.ft.com/cms/s/2/794bbb56-1f8e-11e3-8861-00144feab7...](http://www.ft.com/cms/s/2/794bbb56-1f8e-11e3-8861-00144feab7de.html#axzz2iUmqtKr9)

~~~
Swannie
All outsourced.

GDS is trying to in-source a lot of work that should be controlled by a
central publishing and transaction design group, and contracting out the
relatively boring task of following their rules. Good idea, as long as their
architecture/design/program management review boards are well staffed and
motivated.

I don't think they will stop these large failures. And many of these large
failures do have clear up-front requirements that cannot be changed. Iterative
waterfall is not so terrible.

~~~
hga
Funny thing is, as far as we can tell this is how that project was done, minus
the contract size sorts of splits. The government didn't hire a integrator,
HHS's CMS took on that responsibility including integration testing.

Of course the minor fly in the ointment is that CMS didn't even vaguely have
the expertise to pull this off; the Pentagon can do this for medium sizes
weapons projects (which are a rather different field), but no one else in the
US government has been said to have it.

------
jarrett
I would also add that the project had too many cooks in the kitchen, by some
accounts. I have heard there were upwards of 50 distinct companies
subcontracting on this project.

I work on projects that are probably on par in terms of complexity. We
typically only involve a handful of firms. And even then, coordinating them
all is a challenge. I can't fathom making the process work with 50+ firms.

Maybe that number was hyperbole. I don't know. But if it's true, I shudder at
the thought.

~~~
nraynaud
Yeah, and that's one of the reasons I'm a bit wary of this "open-source is
magic" mantra, that's putting a lot of cooks in the burning kitchen. Open-
source means community management, public relations with opiniatred people,
Linus-grade emails, and if you have really a big participation but no strong
leader, it ends up like GNU hurd (is it dead yet ?).

It's all about organization, and trying to have just enough people to get the
work done and nobody more, and the right people, and this has nothing to do
with open or closed source.

~~~
gohrt
GNU Hurd died because Linux appeared, folks like RedHat combined the GNU
userspace with the Linux kernel.

It was a massive success -- everyone put in the part they did well -- kernel +
userland + distribution = WIN.

~~~
marcosdumay
> folks like RedHat combined the GNU userspace with the Linux kernel

Once Linux was good enough, RMS himself set Hurd aside, put the Linux kernel
into the GNU project that he started, and (almost literally) declared mission
accomplished. It wasn't folks like RedHat (that only came years later), the
developers of Hurd were the first to kill it.

But then, since Hurd has quite an interesting architecture, people keep
developing it, like dozens of OSs out there that'll never get anywhere, but
are fine with that.

------
sien
It's not just the US government that has crazy, weird, inefficient technology.
I'd be surprised if every government wasn't like this. The biggest IT failure
in the world was the UK's attempt at healthcare computing that cost 12Bn GBP
and didn't deliver a functioning system.

I work as a contractor for the Australian government. I personally know of
multiple project failures in the 10s of millions of AUD range and a few in the
hundreds of millions. These stories don't even make the news.

I've worked at small companies, research institutes, universities and now in
government. I've not worked big corporate but have heard that it is similar,
although more efficient than government. Size means you get less feedback on
what is really useful.

Government fundamentally lacks feedback on what really matters. In the US the
department of health cannot be driven out of business by another department
that does what is important 10% better. In private industry that discipline
and feedback makes things work.

If you build a widget X and it isn't something that people want you go under.
That doesn't happen in government. If you build a donkey but it's the donkey
they paid for it could be in service for 20 years.

It's hard to see how to make it all better. Perhaps trying to keep components
small and having multiple groups build them and select the best might help.
Then at least 2+ groups would have to compete to build a better system.

~~~
drblast
I kind of like the idea of every government function has two independent
providers that compete for funding, and citizens could choose which provider
to use.

I'm sure that system would blow up in some other way, however.

~~~
saraid216
> I kind of like the idea of every government function has two independent
> providers that compete for funding

This is what the current government contractor system looks like.

> and citizens could choose which provider to use.

And how would we receive these choices? Balloting is a government function...
Should we ask Diebold?

------
coolrhymes
ex-cgi contractor here and I am not surprised. they did the Massachusetts
health care system that cost over millions of dollars and didn't deliver on
the first day. The government had to shell out even more to keep it running
bcoz Gov. Patrick didn't want that to fail on his watch.

The way they work is purely in water-fall project management mode. Project
managers are gods and spend insane amount of time on ms-office calculating
hours per each task that are 2 years out. then they bring CGI indian sub-par
programmers on L1 to save on costs. Technology is least of their concerns
since its about shipping code. Also blame shouldn't be just on CGI, the
government is at fault as well. Simple request for information would take
about 4 biz days to get it. everything is slow and the Gov IT staff has no
clue on how to scale. Anyway, when I heard CGI won this project, I knew it
would fail.

------
danso
If this is the software that's developed for millions of public users, imagine
the software developed for in-house use, where the users are too few and too
unsavvy to see how the software could be better (this is the case with most
businesses, not just government)...this applies to basic information
processing and to software interfaces for our sophisticated weapon systems.

And even the software for info systems can have dangerous consequences. Does
anyone remember the underwear bomber, who almost brought down a plane and
caused a nice surge of invasive security measures afterwards? His own father
exposed him, but the State Dept's visa system failed to find the terrorist
because someone misspelled his name when entering it into the system

[http://www.cnn.com/2010/POLITICS/01/08/terror.suspect.visa/i...](http://www.cnn.com/2010/POLITICS/01/08/terror.suspect.visa/index.html)

Think about it...the State Dept has been dealing with foreign terrorists well
before 9/11, whose names are easily misspelled by Westerners...there's not
even a consistent way to spell Osama bin Laden, depending on you interpret he
phonetics. And yet no one thought that a fuzzy spellcheck would be useful,
apparently. And a whole bunch of people almost died because of it (and the
security apparatus greatly increased)

~~~
artsrc
Software for in-house use might be fine.

Bureaucracy reduces efficiency.

The more organizationally significant the software the worse risk there is.

Processes and procedures are the ways institutions manage risk.

So the more significant the software is the less efficient the production of
it is.

------
smurph
I've had federal employees tell me with a straight face that waterfall
development is the only model that works, and that is why 'all of the tech
companies use it'. These people have often gone and gotten certifications for
stuff like six sigma and CMMI. They will never change their tune. You
basically have to wait for all of them to retire. The average government tech
worker is so different from the commercial tech worker that they may as well
be a different industry all together.

~~~
gohrt
I once worked for a very successful software company with a customer base that
included government (IT installations), academic (college courses lab work,
hardware research), financial (banks), and industry (computer hardware
manufacturers).

We built a platform, and had a consulting wing that built custom "apps" on top
of it.

We did Waterfall and CMMI. Waterfall was most intense for the consulting
projects. I remember being assigned to build feature 3.2.2.1 in the spec.

------
moron4hire
The problem is not necessarily Waterfall, it's people's unimaginative approach
to it. I've done plenty of projects for clients that wanted a Waterfall
methodology, and I did it by writing the documentation and the prototyping
code at the same time. In other words, Agile fits inside Waterfall. The
requirements gathering phase in Waterfall projects is so incredibly long that
you can definitely afford to make a prototype or 5. And you win huge points
with your client when you're done with the requirements phase and get to say
that development will take "only two months".

You have to treat prototyping as part of the requirements gather process.
Then, when requirements phase is done, you have to treat "development" as
really "testing". Because, for the types of clients that are going to insist
on a Waterfall project, the final testing is really only a cursory user
acceptance testing and they really don't have the skills necessary to
determine if you've met their requirements or not.

~~~
vonmoltke
With the government, depending on how "involved" your customer is in the
contract, they might have a shitfit if they find out you are doing this. As
another poster in this section noted, there are plenty of government and
quasi-government[1] employees who seriously believe that you can't start
writing code until you have defined all your requirements and prepared a
design to meet those requirements.

[1] People who work for companies like MITRE that are basically privatized
extensions of the government.

~~~
moron4hire
You are right, which is why I don't actually do government contracts anymore.
It's just too easy to end up working for a complete, abject moron.

~~~
moron4hire
I should say, this is also why I don't work for Fortune 500 companies anymore,
and also why I don't do work for fly-by-night, no-technical-cofounder
startups, either.

------
jamii
It's interesting that the UK seems to be getting around this by doing stuff
in-house. So now we have the open government license
([http://www.nationalarchives.gov.uk/doc/open-government-
licen...](http://www.nationalarchives.gov.uk/doc/open-government-
licence/version/2/)) and government websites that take pull requests for
content.

~~~
lmm
It's the only way that works. You can contract out parts of your
implementation, but you need a certain level of in-house technical expertise
just to not get taken for a ride by your contractors.

------
bchjam
Does anyone else find it ironic that Obama's campaign was a picture of web
execution but in his administration it's the opposite?

~~~
buckbova
The health care site involves a web application with high number of business
rules and data structures.

The campaign's site is fluff.

~~~
esw
The campaign's site was also _static_ (see [http://kylerush.net/blog/meet-the-
obama-campaigns-250-millio...](http://kylerush.net/blog/meet-the-obama-
campaigns-250-million-fundraising-platform/))

------
fpp
Have a look at what's happening in the UK:

[https://www.gov.uk/transformation](https://www.gov.uk/transformation) (fully
responsive design pages together with new service backends delivered with
Agile / Scrum)

and the teams doing it:
[http://digital.cabinetoffice.gov.uk/](http://digital.cabinetoffice.gov.uk/)
(they are hiring more than a dozen people in the moment)

------
DanielBMarkham
This is not a bad article. The reporter manages to cover a complex subject in
an easy-to-understand way. I liked it.

I will point out, however, that there's a huge assumption lurking in there
that wasn't explicitly stated: _somebody on the government side has to know
what they want and be willing to take the heat if they get it wrong_. _This_
is the reason so many agencies prefer waterfall -- there's enough obfuscation
and paperwork involved that when somebody complains, and in high-risk projects
there'll always be complainers, nobody is really at fault. The coder guys can
point back to the designer guys. The designer guys can point back to the
requirements guys. You'd think that the requirements guys, the guys at the
front of the waterfall, would catch all the blame, and they do. But they just
write bug tickets because some aspect of the process wasn't followed well
enough.

You can spend hours or days trying to figure out what went wrong and not know
anything more than you did before you started. Which is exactly why the system
has evolved the way it has.

I hear a lot more government projects are going to be Agile. Here's wishing
them luck. If done correctly, Agile will 'debug' the organizational problems
that lead to this bad performance over and over again. If they just sprinkle a
little Agile nomenclature on top of things, it won't do anything at all.

------
saraid216
So, for everyone who feels that they have anything resembling a better
solution, I'd suggest that you go and actually try implementing it at a
smaller scale first. Start with your homeowners' association, your
neighborhood, or your nearest town. You'll have relatively few people to
convince, more access to decision makers and funding sources, and less capable
contractors gaming your system.

Get that down, and then get several neighboring communities--again, HOAs,
neighborhoods, or towns--and get them to adopt your ideas as well. With that
amount of variation, you've got a strong base from which you can convince a
major city, or a county, to adopt your ideas: after all, many of their
constituents are already on it and can endorse it.

This isn't meaningfully different from founding a startup taking on
governments as clients.

------
technotarek
The most important part of this article is paragraph two, for without the
exclusivity present in the government procurement process, it's unlikely we'd
have a lack of innovation in regard to development practice.

I've led or worked on tech contracts and grants for ED, HHS, NSF, CDC, and
others. Several people have pointed out some important points that are not
getting enough attention in my opinion:

by @mcone: "The procurement rules were designed for that, yes. But in real
world scenarios, those rules effectively do exactly the opposite. Since there
are so many hoops for potential vendors to jump through, only the most
established players get to bid on most contracts. And in my experience
corruption and cronyism is still alive and well in federal IT contracting."

It's true, incest between government and industry is rampant and has led to
wide spread cronyism despite the system's best efforts to limit the effects.
People that once worked for company X, now serve on the proposal review panels
when company X competes for work. No, they don't receive direct compensation
and thus there is no immediate conflict of interest, but the reality is humans
are drawn to (or don't want to disappoint) the people they know (former
colleagues) and thus pick their old companies. In addition, they know there is
a chance they may once again return to said company (so there is long term
conflict of interest potential).

Another point that hasn't been discussed is how the government's procurement
process provides next to no incentive for companies to efficiently produce
good products. If our industry loves the DRY concept, everything about the gov
procurement process points to a !DRY (or do repeat yourself). We built and
rebuilt the same database for offices of the government that shared a building
with one another. But because everyone is in a silo, they don't collaborate
well and don't realize that they could pool their needs to develop more
universal products. (And on the industry side, as long as gov continues to
work this way, they don't even have an incentive to re-use their work or
propose innovative, generalizable solutions.)

And for those of you that might say, 'but can't you win a gov contract by
bidding lower by working off of existing work?' The truth is price has very
little to do with who gets selected for a gov contract.

------
Systemic33
Maybe US should look to the number of successful websites launched in European
governments and public sectors. There were obviously a lot of failures along
the way, but that is what you learn from. And if US can skip some failures,
thats maybe a few billions not lost. Worth looking into.

------
samspot
I have respect for people who are doing waterfall and admit to it. In my
personal sphere I see far too many people talking agile all day while actually
running waterfall projects. It seems like people think agile means "waterfall,
but skip most of the upfront planning".

------
ethanazir
the DoD does not know what it wants ahead of time; they are in a tight
feedback loop with contractors.

------
jheriko
actual problem: government is incompetent at leadership because it is
saturated with politicians whose main skill is only convincing you they are
competent leaders. no leadership skills actually required.

if government were competent stuff like capitalist economy and privatising
public services would always be bad (provably - actual proof, not just
overwhelming evidence - but there is overwhelming evidence too).

leaders intuit things like agile and don't label them - instead of picking it
up from a book and implementing it badly because they don't understand first
principles.

------
xarien
Coming from a 6+ year stint in a defense contracting, I can safely say that
the issue with this approach happens well before the testing. The problem more
often than not occurs at the requirements level.

~~~
hga
Bingo. The government, which took the role of integrator, kept making
requirements changes, right through the week before launch. They also did
integration testing ... and of course ignored that that failed hard.

I can't see _any_ way CGI Federal et. al. could have won.

~~~
marcosdumay
> The government ... kept making requirements changes

So, just like every client every developer ever had?

~~~
hga
If they were all that bad we'd _never_ get anything done.

In drafting the above I removed some thoughts about CGI Federal pushing back,
since that struck me as something that probably wouldn't work in this context.
But I for one have pushed back on clueless customers before, saying the usual
"this change will cost time, money and/or quality".

Or, heck, I'll bet CGI did some push back, but there's no reason to believe
the inexperienced bureaucrats and political appointees in HHS and CMS would
have listened to them. We've been reliably told they were told it was going to
hell, and there's that one fed who switching in March to "I hope it won't be a
Third World experience" which I take as a sign he saw trouble on the horizon.

------
hga
" _A tank is a tank is a tank, pretty much, plus or minus a few bells and
whistles. "_

Geeze, such amazing ignorance. If you're vaguely interested in this sort of
thing, and want to learn all the process and engineering reasons the Abrams
M-1 became the _King of the Killing Zone_ , get a copy of the book by that
name: [http://www.amazon.com/King-Killing-Zone-Orr-
Kelly/dp/0393026...](http://www.amazon.com/King-Killing-Zone-Orr-
Kelly/dp/0393026485/)

Written by someone who initially expected to castigate it due to early
(mis)reported teething problems (e.g. the whole "it throws tracks (more than
other tanks)" was due to a proving ground's faulty tension meter), he got
completely sold on the tank which has since totally proven its worth.

Lots of fun stuff, from their modeling everything with strong constraints like
weight (i.e. what bridges can it cross), e.g. they didn't want to provide a
heavy M2 .50 BMG but the tankers demanded it. To the successful development
team's leader, a grizzled Chrysler car exec who drove them crazy with "that
doesn't look good" sorts of complaints.

Which often turned out to be a boon (ignoring that weapons should look good so
their users feel good about them, which the M-1 delivers on). Said it was too
high in an ugly way, so they figured out how to shave a foot off, which is
very important for the European theater (not so good for deserts). Didn't like
how the armor skirts didn't extend all the way to the back. So they gave in
(I'm sure the modeling said it was only a minor net loss) ... and found that
made a critial difference in keeping cruft thrown up by the tracks out of its
turbine engine.

Very much an iterative process, in a domain where you truly "bend metal" to
get things done.

So take the author's words with a big grain of salt, she's woefully ignorant
of a huge domain in which we've been building for a very long time the world's
most sophisticated artifacts, and learning how to, and how not to do it ...
with stakes no less than national survival. Digital computers used for IT are
a very recent development as these things go.

------
interstitial
I don't know why websites can't run on magic and fairies. The rest of the
government does.

------
mh_yam
Oh really, ya think?

