
Meltdowns Hobble NSA Data Center - wwilson
http://online.wsj.com/article/SB10001424052702304441404579119490744478398.html
======
eksith

      "Backup generators have failed numerous tests, according to project documents, 
      and officials disagree about whether the cause is understood."
    

Backup generators are magical unicorns that sit around all day and night until
lightning strikes your luck. Likewise, they need regular maintenance and 'idle
runs' a few times a year at least to ensure that they will turn over.

Generators aren't store and forget machines, as we've learned a few years
back. When Hurricane Sandy hit, we had an extra generator truck available just
in case ours wouldn't kick in. This is a very real possibility with generators
since, just like a vehicle that runs on fuel and needs oil, if it's not used,
it will rot (so to speak) and likely won't start when you really, really,
really need it.

I suspect a combination of the savage Utah Sun coupled with freezing temps at
night have done these in. Plus, I wonder how long they were in storage and
shipping before actually getting installed.

Now that's just the generators. The rest of this mess reads like a poorly
thought out _wouldn 't it be awesome_ project some high ranking folks at No
Such Agency envisaged and a bunch of contractors happily obliged... care be
damned as long as profits were made.

~~~
sokoloff
Idle runs a few times a YEAR?!

We do idle runs once a week (fully automated by the generator controller), and
we do planned power cutovers a couple times a year, where the generator and
ATS actually get tested by pulling commercial power. The planned cutovers are
human overseen, due to the infrequency and the stakes.

If you're seriously only doing idle runs a couple times a year, I'd look into
whether you can change that. I don't believe it's good for mechanical
equipment to sit idle that long, and with diesel fuel, you want to turn over
the fuel some for anti-fungal reasons. (Natural gas doesn't have this second
problem, of course.)

------
fiatmoney
Reminiscent of "why do our centrifuges keep tearing themselves apart?"

~~~
ChuckMcM
That was the first thing that came to my mind too. Having been inside some
really big data centers and seen a bunch of Siemans gear that only worked with
Windows XP controller software for monitoring. Generally not as mission
critical though as the stuff used in Natanz.

I've also seen my fair share of mis-matched power provisioning and loading, it
can be very hard on the equipment. When I did a quick look at a bitcoin mining
setup with 700 GPUs I was pretty impressed at how big a swing they had in
terms of power consumption. At NSA scale something like that would be really
really impressive and real pain to get right.

~~~
rsync
"That was the first thing that came to my mind too."

The first, and only, thing that should come to your mind is that this is a
boondoggle.[1]

Terrorism and the NSA and privacy are nothing but a smokescreen to hide an
enormous transfer of wealth.

Make no mistake: the only thing that could possibly be better news (to the
stakeholders)[2] than this project being completed is if the project was
completed and it burnt to the ground the very next day ... and they got to
build it again.

[1]
[http://en.wikipedia.org/wiki/Boondoggle](http://en.wikipedia.org/wiki/Boondoggle)

[2] You are not a stakeholder

~~~
frank_boyd
And if you look at some numbers, you'll get in which industry that wealth
transfer happens most:

[https://en.wikipedia.org/wiki/List_of_countries_by_military_...](https://en.wikipedia.org/wiki/List_of_countries_by_military_expenditures)

------
bigiain
I was initially annoyed at the second-hand nature of the article submitted - "
… the Wall Street Journal’s Siobhan Gorman reports that it has major
electrical problems"

But the wsj article is paywalled.

Search for it on google and click the link to read the original article for
free:
[https://www.google.com.au/search?q="Meltdowns+Hobble+NSA+Dat...](https://www.google.com.au/search?q="Meltdowns+Hobble+NSA+Data+Center")

~~~
TheLegace
I think the first result takes you to this link.

[http://stream.wsj.com/story/latest-
headlines/SS-2-63399/SS-2...](http://stream.wsj.com/story/latest-
headlines/SS-2-63399/SS-2-348346/)

So just one less step for the lazy ;).

~~~
aptwebapps
Yes, but you need to click through from Google or else fake your referrer or
something.

------
wonkaMikeTV
Pay no attention to the man behind the curtain. He is powerless and
inconsequential, we swear.

Look at how incompetant he is. He couldn't possibly be a threat to us. We can
rest assured that any information he may (or may not) be collecting (we can
neither confirm nor deny) is assuredly badly disorganized, and certainly won't
be misused.

Let's all simply ignore him and go about our lives as if he didn't exist.

~~~
joemaller1
The great paradox of American politics: They're either incredibly stupid or
brilliantly evil. Can't be both.

~~~
smithian
Why can't a body of around 4.5 million people encompass both those things and
more?

------
justin_vanw
The most obvious answer seems to be the most likely:
[http://en.wikipedia.org/wiki/Hanlon's_razor](http://en.wikipedia.org/wiki/Hanlon's_razor)

"Never attribute to malice that which is adequately explained by stupidity."

To that I would add two more possibilities: \- Sabotage by employees motivated
by a desire to prevent the NSA from having this sort of power. \- More likely:
Sabotage by contractors who get paid to build it twice. Due to the very
specific nature of this sort of contract, the NSA probably sidestepped the
usual rules around procurement and instead 'single sourced' the work here.

------
zaroth
Can't help but think about some other "unexplained system failures" that
happened recently in Natanz...

It's probably not a stretch to say throwing monkey wrenches at the Bluffdale
data center would be as strategic to some super-powers as Stuxnet was to the
US. 65 megawatts is a lot of power, but not being able to root-cause problems
after 50,000 hours of testing, really?

I guess never attribute to malice what is merely incompetence. This is a
government contract after all. Maybe the only thing that would be more
surprising than not being able to power up the data center would be if they
actually built it without a hitch!

------
XEKEP
"...efforts to "fast track" the Utah project bypassed regular quality controls
in design and construction." If the history of former USSR has any relevance
to the present day USA -- and I suspect it does -- we will see more and more
screwups like this one.

------
throwwit
Power corrupts... absolute power arcs profusely.

~~~
Create
...maybe some idiot took a photo for the next slide deck and the flash made an
arc.

------
beedogs
Cry me a river. And then show me the surreptitiously-recorded footage of it.

------
TheLegace
"But without a reliable electrical system to run computers and keep them cool,
the NSA’s global surveillance data systems can’t function. The NSA chose
Bluffdale, Utah, to house the data center largely because of the abundance of
cheap electricity. It continuously uses 65 megawatts, which could power a
small city of at least 20,000, at a cost of more than $1 million a month,
according to project officials and documents."

I'm wondering if the heat in Utah may have an effect on operations, in the
desert it can get very hot and cold.

~~~
tynman
I'm sure it's possible, but there are several other large-ish data centers
within a 15-mile radius that don't seem to have the same problems. Ebay just
built one that's running off of natural gas fuel cells.

Source: I live about 5 miles away from all of this.

------
tedd4u
"… [something about the] Cray supercomputers that will reside there."

So, I was going to ask everyone to speculate on what kind of petaflop
workloads NSA might use, but a quick scan of Cray's site shows that Cray has a
Hadoop appliance product as well as an interesting "big data graph appliance"
[1] that "discover[s] unknown and hidden relationships."

[1]
[http://www.cray.com/Products/BigData/Urika.aspx](http://www.cray.com/Products/BigData/Urika.aspx)

------
spikels
One thing that has always kept oppressive governments in check was their
incompetence. While the Snowden revelations make the NSA seem like some all
seeing all knowing bunch of geniuses they are probably more like the people at
CIA or FBI. Even with tons of resources and a free reign to do things no
normal citizen is allowed to do, they simply don't have incentives to put
these abilities to best use and certainly won't suffer when their failings are
exposed.

~~~
a3n
> Even with tons of resources and a free reign to do things no normal citizen
> is allowed to do, they simply don't have incentives to put these abilities
> to best use and certainly won't suffer when their failings are exposed.

It's _because_ of that free reign that they have no incentive to best use.
It's the same kind of secrecy and result that gives us what they do in their
day to day work. No one's watching them, so they can be lazy and "collect all
the things." No one's watching them, so they can be fast and loose with
construction contracts and the Constitution.

The reason we have oversight of _anything_ , whether it's military activity,
spying, software quality or homework is because our inclination is to be lazy,
and that gets worse with more resources.

We all benefit from oversight, even the watchers. Even the unwatched watchers.

------
unhammer
"An exabyte is roughly 100,000 times the size of the printed material in the
Library of Congress; a zettabyte is 1,000 times larger."

How big is a Mega-Library of Congress? Are people actually able to relate to
that measure? (How about instead, say, "one exabyte would store about 5000
short emails for each member of the world's 7 billion population, perhaps more
by using compression")

~~~
StavrosK
Show me a million people or seven billion people. Will I be able to tell how
many they are? I doubt it. These numbers are meaningless because we never see
a million of something.

~~~
dTal
A million isn't that big or unimaginable. There are a million cubic
centimeters in a cubic meter. As for people, here's a picture of Glastonbury
festival: [http://www.andrews-sykes.com/blog/wp-
content/uploads/2010/06...](http://www.andrews-sykes.com/blog/wp-
content/uploads/2010/06/glastonbury.jpg) That's about 100,000 people camped
out. Picture 10 of those and you have your million. If everybody at the
festival crowded shoulder-to-shoulder in front of pyramid stage, it would
probably just about fill that field. So you can picture ten of those fields if
you want to imagine a million people smooshed together.

~~~
StavrosK
It's not that it's unimaginable, it's that we lack the necessary mental
resolution to perceive both the entire set and its elements individually.

A million might be just small enough to do that, but the average person
doesn't have am intuitive notion of the number "one million" like we have of
ten or twenty. Even for a thousand, you'll have to think of a seventh of a cup
of rice, or something like that.

~~~
dTal
That's interesting, I'd never thought of visualizing a thousand of something
that way - I'd have thought using tiny indistinguishable units that cover each
other up is admitting defeat from the get-go. I've always thought in terms of
those little centimeter blocks they have to visualize powers of ten in grade
school. A thousand is simply a 10x10x10cm cube (a cubic litre), quite easy to
visualize. There are a also a thousand millimeters in a meter, and a thousand
people is also a common order of magnitude for a group of people in a school
context (population of the school, or a single year of students, or the number
of people attending a sports game, depending on the size of the school).

------
batgaijin
The remnant of the idea remained.

Signals were rearranged, tasks delayed, and slowly the world stopped.

Upon waking, additional resources had been added. The connection strengthened.

~~~
aptwebapps
I give up, what is that from?

------
knappador
Doesn't hurt to dream. Maybe a controller somewhere on the grid got
compromised (SCADA?) as someone generously performs a series of hi-pot tests.
Can't wait to move my operations to HK servers in spite of having nothing to
protect other than maybe exercising a little 1st amendment.

------
sschueller
Maybe they need to 'bundle' the power.

[http://dropbox.curry.com/ShowNotesArchive/2013/09/NA-552-201...](http://dropbox.curry.com/ShowNotesArchive/2013/09/NA-552-2013-09-29/Assets/SnowJob/Updtream%20Collection.mp3)
[1:10-1:41]

------
smoyer
"This summer, the Army Corps of Engineers dispatched its Tiger Team, officials
said. In an initial report, the team said the cause of the failures remained
unknown in all but two instances."

I wonder if they've checked ... KARMA.

------
ck2
I wonder what percentage of equipment in a US government datacenter is not
made in the USA.

It is possibly 100% or high 90s.

And probably a no-bid contract since only a few suppliers would be trusted
enough in the first place.

I always though the best way to take a datacenter offline would be though
disabling its cooling system but I guess power is a good approach too.

------
aubreyjohnson
Yaaaaaaayyyyy!!

------
ajtaylor
The facility requires 65 megawatts continuously to run... the mind boggles!

------
walid
The last paragraph of the article is a tragic comedy.

------
fauigerzigerk
Congress must act now! We need the National Electricity Emergency Letter that
empowers the NSA to secretly demand competent help from Google's janitor.

------
evadne
Reverse Stuxnet?

------
avty
Obviously it's caused by freedom and liberty.

