
Computer Crash Wipes Out Years of Air Force Investigation Records - joering2
http://www.govexec.com/defense/2016/06/computer-crash-wipes-out-years-air-force-investigation-records/129058/
======
RcouF1uZ4gsC
There has been too much recently of government agencies losing data due to
"crashes" or "inadvertent" deletions. For example the IRS, state department,
and now the Air Force.

A really easy way to fix this is to make a law so that if government data such
as this gets lost, then the cabinet level person responsible for overseeing
the agency immediately loses their cabinet level position and is barred from
further work with the federal government.

This would do wonders to align incentives all through the federal agencies
when it comes to data safeguards.

~~~
heimatau
This idea is too simplistic. Although I used to support it. I've changed my
position from yours to 'retrained and demotion' position. Since people make
mistakes, I think being 'barred from further work' is counter-productive and
the government would lose out on many able and competent workers. Heck, this
idea could also be exploited by lower-level workers due to not liking their
higher-up bosses; i.e. a lower-level worker intentionally screwing up and
costing the job of the higher-up boss because of the 'new law' stating the
higher-up is now 'barred' from further govt work. Easy way to take out the
competition going up a ladder.

P.S. I think legislating accountability is an issue the public needs to
debate. In all countries. The US has seen numerous gross negligence/errors
(heck one is too many) in various levels of govt (fed, state, local). I hope
we can come up with a better solution than 'fire them and bar them' and even
mine of 'demote/retrain'. There has to be a better way to legislate govt
accountability.

~~~
PakG1
Indeed, such ideas would create an atmosphere of fear and increase cover-ups,
rather than solve problems. There needs to be a solution that acknowledges the
inevitably of mistakes and still deals with the issue.

~~~
iamflimflam1
Exactly this. How many of us would be up in arms if developers were fired for
making mistakes? You just need to look at all the stories around rm -rf...

~~~
emp_zealoth
Maybe thats exactly what should happen? I mean, whole "security" antics where
companies loose tens of millions of logins/credit cards numbers and it all
ends with "southpark BP apology" People fuck up epically and get a slap on a
wrist with a promotion, I mean, come on

------
PakG1
Sigh. When I was in university, I worked at a computer store one summer that
provided IT services to local offices. One of the clients was the Mexican
consulate. We were hired to do various things, including installing a tape
backup system.

I was installing some service packs at about 5pm. Windows NT4. 5pm, everyone's
gone home, safe, right? Server asks if I want to restart, I say yes. Then a
guy walks into the server room, asks if I had restarted the server. I say yes.
Whoops, OK, I guess someone was working.

Next day, the database was dead. Seemingly because I had restarted the server
while someone was using it. It was a custom database job, and the contractor
who had made it was on vacation in some other country. Here's the kicker.
Usually tape backups have rolling tapes, right? But we hadn't started rolling
tapes yet. So the good backup from 2 days prior was overwritten the night
before with the crashed database. Ugh. And I was due to go back to school in 2
weeks in another city. Ouch. I still feel bad about that to this day. No idea
how my boss ended up.

Never ever let someone who doesn't know what they're doing mess around with
your mission critical systems where the consequences are not recoverable.
Always guide them. Although in my case, it seems I may have known more than my
boss at the time, so maybe wasn't possible for me....

------
dmd
The records were not wiped out by a computer crash. They were wiped out by
gross incompetence.

------
toyg
The joys of IT. Back in the day, to wipe years of records you had to burn or
shred tons of paper in multiple locations. Now you just crash one little
system... Despite 20 years of Microsoft-induced crashes, clear best practices
and yadda yadda, most organizations will still have a single backup system for
any given data repository -- if they have one at all.

And the answer when shit happens is "outsource! Go to the cloud!" \-- so that
local managers won't be responsible when the snafu happens and data is lost
(if it happens to Google, it can and will happen to anyone).

Progress, eh.

~~~
bsurmanski
I would consider it progress.

The only data loss to Google that I was able to find was [0], where a single
datacenter was hit by lightning 4 times in short succession, and 0.000001% of
disk space in said datacenter was lost. And only because the lost data
consisted of recent writes that had yet to be backed up. And only because the
lightning affected the AUX power systems in a way that they have since
committed to fixing (or perhaps have fixed).

IMO, cloud storage is the most reliable way to store data, and Google the most
reliable. I never know if I'm going to drop my laptop and lose all work. Or if
the harddrive in my server is going to give up and burst into flames. But the
great thing about cloud storage is that it abstracts away hardware components
and failures so it's somebody else's problem. And they worry a lot more about
data loss than I do. Cloud data storage is designed with the assumption that
hardware is faulty. Spread redundant copies of data to geographically isolated
datacenters, and you're much safer than some admin of averagejoe.com keeping
his data on a single harddrive in his closet.

[0] -
[https://status.cloud.google.com/incident/compute/15056#57195...](https://status.cloud.google.com/incident/compute/15056#5719570367119360)

~~~
toyg
The problem is not just DR, but anything that can result in data loss. There
was the one where they'd delete stuff you didn't mean to [1], the one where
they got stuff back by the scruff of the neck [2]... i can't be arsed to look
further back, but I'm pretty sure shit happened earlier as well (and I'm just
looking at GMail). And of course there are tons of horror stories about
accounts disabled or deleted for commercial or other unspecified reasons,
because once you put your trust in another entity, that's it, they can do as
they please. Their track record might be as good as anyone's, but it still has
a few holes and people should prepare for it rather than considering "the
cloud" as a magic bucket.

(disclaimer: my life would probably be ruined by losing the gmail account I've
held since private-beta days, because life is short and I'm shit at sysadmin;
but there's a big difference between a lone old geek and large
organizations...)

[1]
[http://www.dailymail.co.uk/sciencetech/article-2548010/Has-G...](http://www.dailymail.co.uk/sciencetech/article-2548010/Has-
Gmail-lost-YOUR-emails-Glitch-causes-thousands-users-accidentally-delete-
messages-report-spam.html)

[2]
[http://www.datacenterknowledge.com/archives/2011/03/01/googl...](http://www.datacenterknowledge.com/archives/2011/03/01/google-
turns-to-tape-to-rescue-lost-gmail/)

~~~
eru
I don't get the problem with [2]? The tape worked exactly as intended.

------
molecule
_> He also criticized the Air Force for notifying Congress on Friday
afternoon, five days after senior service leaders was told about the problem._

Prime time for burying bad news.

------
King-Aaron
I recall an episode of the X-Files where they talked about a 'basement fire',
this reminds me of that. When you hear about a high-level government
department "losing" files, it's usually because they've had an office party
around the paper shredder.

Or something along those lines.

~~~
chipperyman573
>office party around the paper shredder

This phrase made me laugh, but I don't really understand what it means, can
you explain please?

~~~
King-Aaron
"frantically destroying documents" I suppose would be another way to frame
it... But I seem to remember that was the wording on the show, or pretty close
to it.

It made it sound like the staff were running around and being very energetic
in their operation of the paper shredder.

~~~
Bartweiss
It's the 'big government agency' version of that movie scene where the hacker
tosses all their drives in the microwave as the feds kick down the door.

------
jtchang
Really? Like there are no backups? Is this amateur hour?

~~~
stonogo
This is a Lockheed snafu, so yes, it is amateur hour.

Much of the data is backed up, and more of it will be retrievable from
original sources. Maybe not all of it; I'll be very interested to see what
gaps remain when they declare the issue resolved.

~~~
762236
Lockheed is one of those companies that fires senior people to save money.
Imagine how this plays out with turbine blades.

~~~
tamana
Why do companies do this, instead of "pay cut, take it or leave it?"

~~~
stonogo
It's never phrased that boldly. People aren't "fired", their contracts are not
renewed, or they are declared redundant because of the new kid with an
advanced degree... usually just after they've finished training the new kid.

The most common move is to cut the highest paid section of the workforce after
they acquire the service contract from a competitor. All major defense firms
do this, which leads to problems maintaining institutional continuity.

------
taneq
Ah, the great flood of '76\. Thousands of records were lost.

Very convenient, that flood.

~~~
emp_zealoth
Genuine question - can you post some more information?

~~~
taneq
It was a quote from the TV show 'Yes, Minister':

James Hacker: How am I going to explain the missing documents to "The Mail"?

Sir Humphrey Appleby: Well, this is what we normally do in circumstances like
these.

James Hacker: [reads memo] This file contains the complete set of papers,
except for a number of secret documents, a few others which are part of still
active files, some correspondence lost in the floods of 1967...

James Hacker: Was 1967 a particularly bad winter?

Sir Humphrey Appleby: No, a marvellous winter. We lost no end of embarrassing
files.

James Hacker: [reads] Some records which went astray in the move to London and
others when the War Office was incorporated in the Ministry of Defence, and
the normal withdrawal of papers whose publication could give grounds for an
action for libel or breach of confidence or cause embarrassment to friendly
governments.

James Hacker: That's pretty comprehensive. How many does that normally leave
for them to look at?

James Hacker: How many does it actually leave? About a hundred?... Fifty?...
Ten?... Five?... Four?... Three?... Two?... One?... _Zero?_

Sir Humphrey Appleby: Yes, Minister.

(Source:
[http://www.imdb.com/character/ch0030014/quotes](http://www.imdb.com/character/ch0030014/quotes))

Edit: Although apparently a similar real-life incident occurred when Hurricane
Sandy wiped out a significant portion of an FBI record archive:
[https://nsarchive.wordpress.com/2014/09/16/archival-
neglect-...](https://nsarchive.wordpress.com/2014/09/16/archival-neglect-
flooding-of-fbi-archives-destroyed-hundreds-of-thousands-of-pages-of-files-
related-to-civil-rights-movement-history/)

~~~
goldenkey
Reminds me of this: The Front Fell Off -
[http://m.youtube.com/watch?v=3m5qxZm_JqM](http://m.youtube.com/watch?v=3m5qxZm_JqM)

~~~
tamana
The is Australian, and if you like it, look for _The Games_ on YouTube, a mock
drama series about the 1996 Sydney Olympic Games organizing committee

------
paulgerhardt
If you're going to say something about paper, see also the 1973 Personnel
Records Archive Fire:
[https://en.wikipedia.org/wiki/National_Personnel_Records_Cen...](https://en.wikipedia.org/wiki/National_Personnel_Records_Center_fire)

------
metaphor
Dear 18F [1]: Help, please.

[1] [https://cloud.gov/](https://cloud.gov/)

~~~
niels_olson
18F is severely outgunned. They need _your_ help. I can't say that enough. The
entire Western US is on fire, there's a drought, they have one fire hose and
their next boss may turn it off.

~~~
lbhnact
Nice to see an encouraging response for 18F here. They are fighting the good
fight, and most people don't even know what they are up against.

------
Falkon1313
>Data about current investigations has also been lost, which is delaying them.

[...]

>"We've opened an investigation to try to find out what’s going on, but right
now, we just don’t know," Stefanek said.

I wonder where they're putting the files for that investigation.

I wonder if their backups were corrupted too, or if they just overwrite the
old ones with newer ones as a cost-saving measure. Or whether no one ever
tried restoring from them and for whatever reason they never could have been
restored from.

Since people rarely prioritize it until it's too late, maybe 'restore from
backup' drills should be a common thing.

------
ww520
May the additional security requirement deter sensible backup procedure? Like
you can't have copies of backup somewhere? Or backups have to be encrypted and
the keys are in a USB drive somewhere, which had gone missing/bad and the
encrypted data can't be recovered?

------
MikeNomad
Good Grief. For a lot less than an F-35 program, they could install a TSM rig,
with off sight backup.

Ours (at a state university) deals with petabytes of data. We've had to
escalate to IBM's Tier 3 support a couple of times over the past decade, but
we have never lost a file.

------
crystaln
Any information on the contract with Lockheed or the consequences of
catastrophic failure?

------
aminorex
Do please keep in mind that on September 10th, 2001, the secretary of defense
reported to Congress that the department could not account for 2.3 trillion
dollars of spending.

------
FatAmerican
It's becoming clear that all government data should be part of blockchain(s).

------
fit2rule
Just ask the Russians for a backup. And if that fails, the next-higher
authority: the NSA.

------
jacquesm
Always hard to tell the difference between gross incompetence and malice.

------
HackerPschorr
Oh, _come_ on. Nobody ever made any backups?

------
sklogic
Any sufficiently advanced incompetence is indistinguishable from a malice.

------
oval_atom
Yeah this is believable. Where I worked, in the secured network there were
maybe 15 workstations with a maximum of maybe 5 people working at any one
time, unless there was a "TOUR" going on. We had 2 contractor and 1 government
IT person supporting this network.

I asked the one of the contractor ITs when was the last backup. Response: "Do
you want me to do a backup of your workstation tonight?"

WTF!!! I am no IT slob but the minimum is Mon-Thur backup of new and changed
files that day, and Friday night was full network backup.

Why wasn't this done?!!!! The tape backup makes too much noise. I guess the
bitch couldn't read her endless Romance Novels because of the noise.

I find an intern, not a full employee, working on a design file in the account
of another engineer. WTF!!! We believe that since the intern was leaving soon,
it would appear that the design was the work of the engineer.

Security violation!!!!! The contractor IT people swore they did not know this
was going on...... BS!!! With so few on the network at any one time, and in
the mornings only the intern was on the network, these contractor IT people
were unaware!!!!

The government IT person knew, but she did not want to report it because she
was well aware of the contractor's history of retaliation.

BTW..... during the investigation, the government engineer asked where was it
written that they could not share passwords!!!!! Can you hear Snowden laughing
at this.

All swept under the lumpy rug....... the violations would make the
organization look bad.

As for me..... I got the IR treatment.... that's Isolation and Retaliation...
then after 2 and a half days, supervisor sent me an email as to my where
abouts.... my reply: Retired as of 4:00 PM PST, 3 days previous.

~~~
icebraining
[http://wiki.lspace.org/mediawiki/Multiple_exclamation_marks](http://wiki.lspace.org/mediawiki/Multiple_exclamation_marks)

