
Sinkhole of bureaucracy - AndrewDucker
http://www.washingtonpost.com/sf/national/2014/03/22/sinkhole-of-bureaucracy/
======
incision
I'm painfully familiar with some of this.

My favorite example of just how dysfunctional it all is was an agency which
made a small fuss about how they were modernizing by allowing online job
applications 10+ years ago.

Behind the scenes the applications were simply emailed to a shared mailbox
then printed out and re-entered by hand into the actual position management
system.

The position management system was so dated and overloaded that the folks
entering data had learned how long they could expect to wait (measured in
minutes to tens of minutes) when moving between a given set of screens. They
would submit one screen then work on something else.

Eventually, the agency spent an enormous amount of money implementing some
combination of systems from SAP to consolidate / replace everything.

Now the same folks who were retyping resumes are cut and pasting them into a
series of webforms and emailing around spreadsheets of all the information
that the system lacks forms for.

Hiring managers who had little or no input on requirements for new system
don't even log in, they request printouts from HR.

\---

Having seen things like this play out a few times now I have a few ideas about
what could help one of these projects actually succeed:

* Generate or at least inform requirements with honest information from people at the ground level. Those people toiling away filling up manilla folders and doing data entry know the actual process. The farther up the chain you go the story will move away from reality toward what's effectively nothing more than the way things are supposed to work.

Problem is, the layers of management don't want to allow that kind of access
and will retaliate against their subordinates for being honest. Also, those
low-level people are likely aware enough to know that they're talking
themselves out of a job.

I think the former requires a powerful and attentive enough personality
leading the effort to simply bulldoze the managers. I wonder if the latter
can't be addressed by buying everyone out with generous severance or early
retirement. It would puts a cap on the cost of these broken jobs and processes
rather than continuing to cycle people ad money through them.

* As a rule, the technical and procurement staff of these agencies aren't competent to generate requirements or evaluate what's produced when dealing with outside contractors and implementers. At the same time, the agencies don't pay enough or move fast enough to maintain a competent internal staff.

This is really hard. Hiring outside experts seems to actually work against the
goal as they will invariably work more for some particular vendor than the
agency.

Big companies absolutely collude and actively work to hold down smaller
competitors so "competitive" anything doesn't really work.

One idea I've been kicking around is a sort of a open source template for
government system/project requirements. It would be a easy to follow "test
suite" which proposals could be compared against. It would raise flags or fail
plans and terms which don't meet the standard of contributors with domain
knowledge.

I'd like to write more on this, but I've got a few things to handle this
evening.

~~~
zyxley
The 2010 US Census was a good example of this sort of dysfunctionality (I
worked in one of the offices for it).

Hiring anyone for field jobs was a process something like:

* field operations clerks arrange application/testing sessions and get all applications in on paper

* administrative clerks (other side of the same office) enter all applications into the computer

* when the people running the field operations want to hire field employees, they fill out paper forms with manually selected criteria (local area, languages spoken, etc) and give them to the administrative fiefdom

* administrative clerks enter the form criteria into the computers, print out a paper list of matching applicants

* administrative clerks call down the list until they have enough who have said 'yes' to the job to match the form

* administrative clerks hand the paper lists with selected applicants on them over to the field operations fiefdom

* field operations clerks try to organize the data in an actually usable way, usually by manually entering the lists into Excel or Access

The most bizarre part was the "manual hiring forms" thing. Beyond the cargo-
cult hiring practice of the forms literally only existing so that they could
be filed away somewhere national in case of cover-your-ass legal issues, the
Census didn't actually give us access to any resources for demographics... and
so the only reason my area had anything like representative employees
(important for getting into gated communities, dealing with different
languages, etc) was because we had a set of people spend a week just comparing
paper county tract maps supplied by the feds with locally-supplied city and
county demographic info and writing up dozens of forms with criteria tuned for
specific areas.

------
lambda
So, I've had a tendency to roll my eyes at the "agile" movement, and how much
faith people put in it that it must be the best thing since slice bread. I've
frequently said that a lot of it is just common sense, some of it is just
needless process that just happens to be different, not worse or better, than
traditional processes, and there are a couple of good points that agile
methodologies have brought about.

But then I look at something like this, and I realize that common sense isn't
necessarily so common in a big bureaucracy like this. They have tried three
times to automate this, and failed each time? They've spent 7 years on one
attempt, with absolutely nothing to show for it? I can't imaging going seven
years without some kind of incremental release that you can use to gauge
whether it's working or not.

I mean, why do they seem to think that it's necessary to solve the whole
problem in one big, opaque project? You would think they could do it
incrementally, automating the easiest pieces or automating one step of the
process while leaving a fallback for manual processing for the more complex
cases or the other steps.

~~~
ams6110
The counterintuitive thing that many people don't understand about government
software projects is that failure is the norm. There are relatively few "big"
consultancies that win these projects, and their numbers include Oracle, IBM,
Accenture, Computer Sciences Corp, SAIC, Booz-Allen Hamilton, and a few
others.

Their goal in government procurements is not to deliver working software, it's
just to win the contract, and then staff it with as many people as they can.
We've known since _The Mythical Man-Month_ and other books such as
_Peopleware_ that high staffing levels does not help most software projects,
in fact it hurts them. But if the project fails, which most do, they all still
get paid. And they will still win other contracts. Delivery of working
software is just not high on the priority list--winning contracts and staffing
them is.

There are all the usual problems with big software projects, and add to it
that most government procurement is still based on the experience of building
physical things (jet fighters, ships, highways, etc.) which really doesn't
work for software.

But the big reason governments spend millions of dollars and waste years of
time on software projects is because the contractors are not incented to
deliver working software. In fact their incentives are to deliver failed
projects, so that they can bid on them again.

~~~
mattmcknight
Even crazier is that the government Often creates these situations by
requiring that bidders bid the same number of positions, and then picks the
one with the lowest rates. Lowering cost with fewer, better people just does
not compute to them at all, because they have tried that strategy and been
stuck with fewer just as bad people in the past. They then just use these
people as staffing resources and attempt to manage the delivery themselves.

The major companies in the space (including many you listed) are trying to
push incremental, iterative delivery, but the customers won't get out of their
own way long enough to let it work.

------
acdha
It's easy to blame government bureaucracy but the real problem is the
combination of lack of empowerment and policy which discourages hiring
qualified staff in-house. Most agencies have been facing budget cuts for
awhile and there has been congressional pressure against hiring staff because
hiring contractors allows you to campaign on having shrunk the government, you
can receive donations from contractors, and few reporters will actually do the
homework needed to note that "saved" positions tend to cost more once you
factor in overhead.

Even if you do get permission to hire techies, the GS pay scale limits who
you're going to be able to hire. There aren't all that many jobs at the top
end of the scale and even fewer will be non-managerial spots which means that
your sales-pitch for putting up with bureaucracy and political meddling starts
with a significant pay cut for high demand professions. (This is the same
reason why regulatory capture is such a problem – we'd be much better off
giving the SEC staff 400% raises than having them make decisions about likely
future employers when they get tired of below-market pay)

If you do manage to make it that far, you can do some interesting things. I
know the CFPB runs a far more modern IT shop because they were able to start
without years of legacy-blinkered culture and hire skilled staff directly
rather than relying on contractors through a procurement process which
basically legally binds you to the least functional waterfall process. It'd be
really good for the entire government if that became the model in the future –
hire good people, give them autonomy to get projects done and responsibility
to deliver.

------
al2o3cr
"“I used to chase people for months — literally — for one signature on one
piece of paper. You want to talk about an egregious waste of taxpayer money?”
recalled one worker who left the mine recently and declined to be named
because of fears of retribution."

And of course if they _didn 't_ chase down every signature, we'd be reading an
otherwise identical article from WaPo whining about "wastefraudandabuse".

~~~
rodgerd
Exactly. I work in a government town (although not for the government), and
the fear of "gotcha" is the single biggest inhibiting factor for getting
things done. There are hordes of politically-motivated eyes waiting for the
smallest chink in your armour, so they can blow it into a huge story. It
literally doesn't matter what people try to do - if there are checks and
balances it's "waste"; if they try to do something innovative and interesting
it's "recklessness" and so on.

The people have the government departments they deserve.

~~~
Aloha
Yeah, I second this, I've learned to phrase questions in such a way when
dealing with government employees, where they can give approval, without
having to either say yes or no - in other words phrase it in such a way that
they dont have to say no - No with the government is the default answer.

------
nostromo
I'm both horrified and comforted.

"The only thing that saves us from the bureaucracy is inefficiency. An
efficient bureaucracy is the greatest threat to liberty." -Eugene McCarthy

------
rayiner
> The root of the problem, he said, was that the system had trouble
> synthesizing information from so many sources and calculations based on so
> many laws.

So the gist of the article is that we pay these folks to do things with paper
because the computer based replacements aren't flexible enough to handle the
problem.

~~~
revelation
I can certainly imagine how that went. A monstrous Java Enterprise deployment,
with thousands of classes and many more interfaces, and illustrous names like
IRetireeAnuniatingSpecialCaseBenefitCalculatorSourceSinkBean.

Maybe the _whole problem_ is too complex to be able to solve it in one go, but
thats why we abstract complexity away and return to solving the simple things,
such as reliable digital document storage. Just digitize papers as they come
in or are retrieved from "cold storage", and over time you have enough in the
working set to get most of these poor people out of the mine and into some
sunlight, jesus...

------
asdfologist
This is exactly why many areas of government should be privatized. With zero
competition and virtually no accountability, these bureaucracies have no
incentive to become more efficient.

~~~
acdha
If you read the article again, notice how the fast majority of those millions
of dollars have gone to private companies who failed to deliver working
systems. Each of those companies passed multiple rounds of competitive
bidding, too. The government staff processing things by hand can at least
point to a ton of successfully processed requests rather than a complete
write-off.

Outside of shallow libertarian magical thinking, privatization isn't a
panacea. Anyone who has worked for or with a large corporation has stories
which are at least this bad — the main difference being that you rarely about
about the failures unless it leads to a major lawsuit. The usual argument is
that you aren't required to do business with those companies but … just try to
live without insurance, telecommunications, cars, etc. Markets allow you to
select the least-bad option but it often takes regulation or changing the
problem to make significant improvements. Before someone mentions Google:
consider how happy you'd be if their QA-in-production / no live support
mentality was applied to your retirement checks rather than your free email…

There are two real problems here: the first is the well-known fact that we're
still figuring out how to deal with large-scale IT projects, which affects
everyone, and the fact that government agencies are significantly blocked from
doing things the right way. For the last couple decades, politicians have
“shrunk” government by giving jobs to the private sector rather than hiring
lazy government workers. What this actually means is that instead of hiring
the same person directly they're hired at a significant markup – equal or
greater pay plus the overhead costs of all of the contracting company's staff
and profits – and all of the work is pushed into a procurement process which
forces you into the worst possible form of waterfall.

You're not the first person to notice that things aren't efficient – and I'm
certain that staff at OPM would agree, given how many of them were quoted
saying just that. The solution is easy – allow them to hire good people in
house at market rates and give them the needed support / autonomy – but that's
politically dicey. This and your proposed solution are both likely to fail as
long as Congress has a sizable contingent of people who benefit from problems,
even those created by their policies, and everyone receives donations from
companies which do business with the government. This also won't magically go
away with privatization, as anyone who has seen large deals go to the company
which bought the best lunch can tell you, but at least in the case of the
government the details are public knowledge.

~~~
ef4
> If you read the article again, notice how the fast majority of those
> millions of dollars have gone to private companies who failed to deliver
> working systems. Each of those companies passed multiple rounds of
> competitive bidding, too.

Those companies didn't fail of course, they were wildly successful at raking
in millions of dollars. It's the government that's responsible for spending
that money well, and it's the government that's failing at it.

> The solution is easy... but that's politically dicey.

Which is another way of saying the solution is not easy.

~~~
acdha
> Those companies didn't fail of course, they were wildly successful at raking
> in millions of dollars. It's the government that's responsible for spending
> that money well, and it's the government that's failing at it.

This is true but misses the point that the government really needs to develop
in house talent. If you looked at every one of those contracts, I'm sure you'd
find that part of the work to be performed was project management, design,
etc. but the problem is frequently that the government isn't allowed to hire
people who can evaluate the options. It's quite easy to make a project fail by
giving the client exactly what they asked for…

> > The solution is easy... but that's politically dicey. > Which is another
> way of saying the solution is not easy.

True – we're basically debating semantics but I would argue that the solution
to this problem is easy but that's because the real problem is something else.
As a country we tend to waste a lot of time talking about problems which are
the product of other problems we've chosen not to deal with.

------
bhewes
If anything this shows there is a ton of low-lying fruit for the Feds to pick
to improve operational efficiency. This along with the Pentagon's inability to
do accounting should allow for big improvements. And due to budget constraints
and more government services coming online (Healthcare) there might actual be
a reason to push through some of these changes. Or at least that is my hope.

------
evolve2k
'Work in Progress' is a key form of muda or waste in lean philosophy. The key
is to eliminate waste coming into the system. Employees need to be empowered
to, send straight back incomplete paperwork and to update the forms &
instructions coming into the system that are contributing to the waste.

Considering the gravitational mass of all these processes I would work on
bringing lean manufacturing approaches in first before deriving a computer
based solution.

As per all these things its going to take a lot of leadership to drive any
type of real change.

Ref:
[http://en.m.wikipedia.org/wiki/Lean_manufacturing](http://en.m.wikipedia.org/wiki/Lean_manufacturing)

------
adamwong246
These organizations seem almost organic, an emergent phenomenon that protects
it's own existence. How _does_ one kill a bureaucracy? It's times like these
I'm tempted to throw my support behind the "Eric Schmidt for CEO of America"
stunt.

~~~
AndrewDucker
This one seems to be trying to computerise, in order to be more efficient.
It's just that every time they've tried, they've failed.

~~~
Ma8ee
THe article didn't say much why they failed. There might be some subtle
sabotaging going on.

~~~
mschuster91
The main point is the sheer mass of regulations and laws affecting and quite
possibly contradicting each other.

Essentially you try to make a system that "fits all cases"... with every
possible permutation. Bound to fail, actually.

~~~
kiba
Congress don't do sanity check. They just add new features to a hairball of a
codebase, multiplying the bugs and complexity of the system.

------
drdeadringer
> "What is modern?"

Oh, well, let's not be too hasty in becoming efficient and up-to-date here.
Obviously, having a sneaker-net is fast enough for you simply because the name
has the term "net" in it. One of my old employers has a few vintage DEC VAXes
if you want to start out on saving on your pencil budget.

I can confirm, though that the "chase the physical signature" game is still
alive and well. /shiver

I would, however, like to be the pizza guy with a security clearance.
Employment while it lasts :D

------
ballard
This is how I imagine this department operates:

[http://youtu.be/fKRV40LZSuY](http://youtu.be/fKRV40LZSuY) (Brazil)

------
FollowSteph3
I wrote an article about this having lived through it for a shilled called Why
a hammer costs $5000? And it really all comes down to motivations being very
badly lined up trying versus just doing the same. The link is:
[http://www.followsteph.com/2011/11/22/why-does-a-hammer-
cost...](http://www.followsteph.com/2011/11/22/why-does-a-hammer-cost-5000/)

------
hindsightbias
For $55M a year, this is cheap. The mistake was to repeatedly spend 100's of
millions to try to automate it.

Go take a 35 yr old SF171 or such and scan, OCR and verify it. Get back to me
when you can do that cheaply 10M times.

~~~
acchow
A 61-day processing period is not cheap for the person waiting for money.

You want to digitize old forms for retired people? This is entirely the wrong
approach.

New data should be in digital form. Not even all new data, just one vertical
at a time - new payroll data, new business expense data, new insurance claims
data, etc. This shards the work into tractable, testable, measurable units.
There will be a (long) period when processing retirement forms will need to
pull data from both analog and digital sources. That time will eventually
pass, and the analog gradually phased out.

~~~
watt
Are you aware of time periods involved, though? When you say "new data" do you
understand that this new data will only become pertinent in about 50 years,
when the employee that just came in to system, retires. 50 years later. And
meanwhile, you are processing paperwork for people that entered the system 50
years ago: on today, 1964.

(Ok, maybe closer 40 years, but the point is the very long storage time.) Even
if all new data is digital today, what about 50 years later, how easy will it
be to use digital files that were generated 50 years earlier. What if file
formats change every 10 years. Then 50 years on, you got 5 different file
formats you must be able to work with.

------
Mz
As a minimum, I think they could scan papers into a document system. We had
this when I was in insurance. It still required review by a person but most
files were digital.

This basic thing already exists. I don't understand why it is not being used
for the federal retirement system. Insurance is probably just as bad as the
retirement system: Policies have to comply with state laws in fifty different
states and there are all kinds of SNAFU things that go on because of it.

