
How the Pentagon's payroll quagmire traps America's soldiers - turar
http://www.reuters.com/article/2013/07/09/us-usa-pentagon-payerrors-special-report-idUSBRE96818I20130709
======
m0nastic
This is both awful, somehow unsurprising, and COBOL is a piss poor scapegoat.

I've talked on here in the past about how crazy it is that some of my
customers had huge financial systems that were written in COBOL, quietly still
plugging away.

The most important of which is a system which processes billions of dollars of
transactions a day, and has done so since the early 70's (when they purchased
the what was then off-the-shelf software). You can imagine what a maintenance
nightmare it must be (both because it's old, very important, and written in a
programming language which predates the cotton gin).

Assuming that they can't just rewrite the system (which might be a reality),
they can certainly sit down a bunch of developers and have them learn COBOL.
COBOL isn't even a particularly difficult language, we were writing pretty
decently-sized programs after a few months in high school (and if the system
is that old, it's probably not even using any fancy language features).

If only the military had some means by which they could take people without
past exposure to some skillset and train them to be competent at that
skillset. Maybe part of that training could involve having to climb over a
wall, or some sort of rope course.

~~~
groby_b
It might not even be an issue of training COBOL programmers. I'm aware of more
than one large-scale legacy system that people "somehow" lost the source code
for.

Which means fixing bugs means disassembly, hot patching, and keeping your
fingers crossed. And if you're lucky and the place is incapable of learning,
they'll then lose all documentation for the patch, too. Fixing anything is a
nightmare. On the upside, if you can stomach it, there are _nice_ consulting
fees to be had.

(I'm not aware if the Pentagon lost source code - my experience is with
civilian systems)

~~~
derefr
> Which means fixing bugs means disassembly, hot patching, and keeping your
> fingers crossed.

This sounds like something with extremely high overhead, that would benefit
from scale. That is to say, instead of trying to reverse-engineer and patch
each module every time you need to make a change, just:

1\. run the entire project through a decompiler, making sure that each
generated code file on its own recompiles to something that works when linked
back together with the rest of the existing modules;

2\. and then use that decompiled result as the new codebase, and start
refactoring.

~~~
groby_b
In a reasonable world, yes. In a world where IT is a cost center, this is what
happens:

1) People work around it as much as possible.

2) When the pain is unbearable, you bring in a high-priced consultant, because
your engineers say something about "recompile" and "refactor". Which you as VP
don't understand, are scared of, and are not willing to spend the money on.

3) The consultant puts in a week or so to hack things together. It's much
cheaper than the recompile/refactor thing, and so you as VP are a hero to your
bosses.

4) Since the consultant is paid hourly, documentation seems kind of pricey.
Let's skip that.

5) Lather. Rinse. Repeat.

I _wish_ this was just Dilbert, but in quite a few companies, that's reality.

------
xradionut
Problems with pay were one of the primary reasons I didn't reenlist. This was
an issue in the 80's. I lost days of personal time talking to yeomen, chiefs
and officers just because every other check was late or missing. Turned out
for the best, since I would have ended up on the USS Stark since my current
ship was going to be decommissioned.

~~~
greenyoda
For those who don't remember, this is what happened to the USS Stark in 1987:

[https://en.wikipedia.org/wiki/USS_Stark_(FFG-31)#Missile_att...](https://en.wikipedia.org/wiki/USS_Stark_\(FFG-31\)#Missile_attack)

------
EliRivers
Readers may be interested that Britain's joint armed forces pay system, part
of JPA (Joint Personnel Admin) is also recognised to be something of a
clusterfuck (although not quite as bad now as it was back when it arrived),
dragging in its wake myriad unlucky sods who found themselves paid incorrectly
and struggled to get anything fixed, including an RAF chap who ended up simply
suing the MOD for the money they owed him but found themselves unable to
actually give him.

JPA was put together principally by the no-longer existent EDS, whose
reputation for being able to handle large complex systems is well known in the
industry.

It's worth mentioning that in the industry, smart cookies recognise that
payroll is very much a wicked problem, and payroll for something as
complicated as the armed forces is massively complex.

Less smart cookies do not recognise this and routinely massively underestimate
the work involved and the support required.

~~~
makomk
Technically EDS still exist - they were bought out by HP and are now HP's
Enterprise Services division. Their reputation was also entirely deserved.

~~~
coldcode
We have HP contracted support at work (formerly EDS). We call them
alternatively Helpless People, Hopeless People or High-Priced People.

~~~
rbanffy
I had a similar experience with Accenture. We called them Accidenture...

------
mratzloff
It makes me furious that no one bats an eye at spending a BILLION dollars on a
single presidential campaign but they can't sort out pay and benefits for our
armed forces and veterans.

Surely a team with a background in large systems could take a set of
requirements and produce a working system in 12-18 months.

~~~
jjoonathan
> could take a set of requirements

I think that's the key problem: nobody has a complete big-picture idea of what
the program is doing. Sure, there are people in each service branch that are
familiar with subsets of the overall technology, but I suspect that at the
overlap of these subsets there is significant disagreement over interpretation
& implementation that has never become a visible problem because nobody runs
consistency checks.

Since compiling the mutually contradictory payroll policies into a non-
probabilistic master policy would immediately shed light on these issues, it
would be a political minefield. Every choice you make would be the wrong
choice and nobody is simultaneously high enough on the totem pole to say "live
with it" and low enough on the totem pole to actually implement it.

~~~
gaadd33
Paychex seems to handle payroll policies for a huge range of businesses
without too much trouble. Also I'm not quite sure why the DoD would have
several different payroll policies? It seems like everything is determined by
the relevant laws, similar to how you can look up the pay frequency and amount
of pretty much any pay grade.

~~~
Retric
Laws often say things like 'significant' which are generally interpreted at
the agency level. The DoD has a lot of agency's so these interpretations are
unlikely to line up.

------
speeder
Previous discussion:
[https://news.ycombinator.com/item?id=6022247](https://news.ycombinator.com/item?id=6022247)

------
D9u
I thought that I read this article here once before... But a search turned up
nothing relevant.

The story pisses me off to no end. The fucking Pentagon pisses away TRILLIONS
of dollars yet they can't even get deserving soldiers their just pay?

It's completely unacceptable!

Take money from the NSA to pay these people.

------
riggins
this made me think of an editorial I read during the Iraq war (maybe someone
else will remember the author's name).

The author was a university professor who was urging his son not to join the
military. One of the points he made is that if his son was injured the
military would not take care of him. It was tragic ... the son ended up
joining anyway and was killed in Iraq. I remember feeling like shit when that
happened. Just kept thinking how awful the Dad must have felt.

I can't quite remember the fathers name .. but I'm pretty sure it was
Scandinavian. I keep thinking Uwe S.....?

------
jrarredondo
Two things are striking: \- The amounts under debate are small \- The DFAS
agency's budget is huge (1.36B)

Why not eliminate the agency altogether? If we divide $1.36B by the number of
active military members 1,429,995, that results in $951 / active military
personnel.

If we assume payment issues happen to 10% of the military, then that means we
can just institute a rule that would give every soldier up to $9,510 in free
"payment error insurance."

But I guess having a government body of 12,000 people with the task of giving
soldiers a hard time makes more sense to some federal bureaucrat.

~~~
gwern
> If we assume payment issues happen to 10% of the military, then that means
> we can just institute a rule that would give every soldier up to $9,510 in
> free "payment error insurance."

Sounds like another scandal just waiting to happen.

------
runarb
A sad read indeed. Unfortunately this is not new. No country ever loved there
soldier as much as their soldiers loved their country.

I remember reading about pay issues in the roman army when I was at the
university. It hasn't changed in 2 000 years. Apparently the roman soldiers
experience that their families back home didn't get their pay, and wives was
given receipts where there was subtracted xx day for "leave" because their
husbands was dead at the time.

We can only speculate if there is any motive and reason her, or only
incompetence…

------
swamp40
> _This agency, with headquarters in Indianapolis, Indiana, has roughly 12,000
> employees and, after cuts under the federal sequester, a $1.36 billion
> budget. It is responsible for accurately paying America 's 2.7 million
> active-duty and Reserve soldiers, sailors, airmen and Marines.

It often fails at that task, a Reuters investigation finds._

Ripe for disruption, says I.

------
jmah
I heard the same story on NPR a few weeks ago (interview with the reporter):
[http://www.npr.org/2013/07/16/202360167/investigation-
reveal...](http://www.npr.org/2013/07/16/202360167/investigation-reveals-a-
military-payroll-rife-with-glitches)

------
nasmorn
Just imagine Walmarts chief accountant would request 17 billion to modernize
the payroll system. The org is of comparable size.

------
auctiontheory
Shameful.

