

Bucket Brigade Hauls Diesel Fuel up 17 Stories to Keep NYC Data Center Online  - 1SockChuck
http://www.datacenterknowledge.com/archives/2012/10/31/peer-1-mobilizes-diesel-bucket-brigade-at-75-broad/

======
JagMicker
Maybe folks will start to realize that an EDG (Emergency Diesel Generator)
without critical support systems is useless.

I was just reading about the Fukushima accident, and how most of the EDG's
failed because they were water-cooled, and while the EDG's were located out of
harms way, the pumps for providing cooling water were located on low ground
and were damaged by the tsunami. <http://fukushima.ans.org/report/accident-
analysis>

Yesterday I saw a story about the Datagram data center in NYC having to shut
down. From their reports about the damage (<http://www.datagram.com>):

"As of 5pm on October 29, 2012, Datagram had thoroughly tested its emergency
systems at 33 Whitehall, NYC fully staffed and awaiting the storm to hit
Manhattan's shores. Once ConEd lost power to Lower Manhattan, Datagram's
emergency systems kicked on maintaining power to Datagram's datacenter.
Unfortunately, within a couple hours of the storm hitting Manhattan's shores,
the building's entire basement, which houses the building's fuel tank pumps
and sump pumps, was completely filled with water and a few feet into the
lobby. Due to electrical systems being underwater the building was forced to
shut down to avoid fire and permanent damage."

It's pretty obvious that, despite all the disaster planning done in the past,
Datagram (and TEPCO, and others) have really neglected to appreciate the
potential modes of failure for their backup power systems. In both cases, they
misunderstood the threat to critical backup power infrastructure. If your EDG
is on the roof but the fuel pumps and electrical switchgear is in the
basement, what will happen during a flood? If your EDG's are on high ground
but the pumps to cool them are not, what happens during a tsunami?

~~~
tghw
Unfortunately, physics dictates this setup. To get to a top floor generator
(which is generally necessary for ventilation), you have to pump from the
bottom. For most buildings, that means from the basement, where the fuel tanks
are placed for other obvious reasons. (Consider if one started leaking, would
you want it dripping down to all of the floors below it?)

These data centers are well designed, but it's impossible to cover every
disaster scenario.

It's easy for you to sit here after the fact and snipe at them. If you really
think you can do better, go start designing them yourself. If you succeed,
you'll do quite well for yourself.

~~~
DannyBee
Your one complaint about diesel fuel dripping is not a big deal in the short
term. Diesel fuel is quite safe and hard to ignite. THere are plenty of
places/situations where significantly more toxic substances are stored in
tanks that are not in basements. Your logic would apply more there (could you
imagine if acetone was dripping from the ceiling?). As a practical matter,
_anything_ leaking through floors is really bad, but the problem is solvable
and not a good reason to keep fuel tanks in a basement.

So what other reasons are fuel tanks placed in basements that outweigh the
risk of putting them in the basement (everything except tornadoes would seem
to be a con for basements)?

Additionally, "Flooding" would seem to be a fairly typical disaster scenario.
Again, what does putting them in the basement prevent against that makes it a
good alternative.

Plenty of people are designing better datacenters and using them, there is no
need for the parent to go out and do anything. If customers hold internap/et
al accountable, the problem should take care of itself.

(of course, they won't, but that's another matter :P)

~~~
HeyLaughingBoy
It's easy to say diesel fuel is "quite safe and hard to ignite." But if the
NYC Fire Code says you have to store it in the basement, then guess what? _You
have to store it in the basement_

~~~
DannyBee
Then either

1\. Stop claiming they are disaster tolerant, because they aren't. Flooding is
a common case.

The exact rule i'm aware of is "406.5.3 Storage tanks. Motor fuel storage
tanks shall be installed below ground, except as authorized by the rules of
the Fire Department"

So get the fire department to authorize it.

2\. Datacener folks should lobby new york city. Bloomberg would likely care.

3\. Find something other than motor fuel, which is what is restricted :)

If all else fails:

4\. Stop building disaster tolerant datacenters in new york. This is why a lot
of financial folks who aren't HFT's build in Seacaucus, etc. There are very
few people who _need_ to place a DC in new york, and can't afford to be across
the river.

(I'm very keenly aware of the bandwidth situation in both places, FWIW)

I'm sure if datacenters start moving across the river, Bloomberg would care.

Building codes and other restrictions _are_ taken into account by the big boys
(Google, Apple, Facebook, etc) when they build large datacenters.

------
derfniw
Makes me think about <http://xkcd.com/705/> .

------
fr0sty
Here is some information on the setup @ 75 Broad St:

"The 17th and 18th floors of 75 Broad have been reserved for generator farms
that can accommodate as many as 40 machines. Big doors will be installed in
the facade of both floors so the generators can be rigged into the building.

A 41,000-gallon fuel tank is being installed in the basement, with a separate
generator and three redundant pumps to supply the generators on the 17th and
18th floors. Each tenant will own its own generator -- E-Spire already has one
installed outside on the setback on the 17th floor -- but the building will
sell them fuel."

[http://www.nytimes.com/1999/10/10/realestate/commercial-
prop...](http://www.nytimes.com/1999/10/10/realestate/commercial-
property-75-broad-street-turning-buildings-into-
telecommunications.html?pagewanted=all&src=pm)

------
dfj225
Squarespace is one of the companies in that data center. See here for their
story (and some photos):

<http://blog.squarespace.com/> <http://status.squarespace.com/>

~~~
adanto6840
Thanks for the link! It had exactly what I was hoping for from the actual
article / OP.

------
bjornsteffanson
This shows great dedication on behalf of the team to provide a temporary
solution to a more permanent problem. Well done.

More importantly, though - and not to discredit any of the hard work that's
been done - hopefully the companies take a look at why the problem was created
in the first place. For instance: why were the generators on the 17th floor?
Why were the pumps below ground? Why was the datacenter built in a floodzone
in the first place?

This is not unlike a lot of problems we face in software - developers bearing
the consequences of poor planning.

------
minikites
I use Fastmail.fm for email and they're hosted at NYI, which seems to be fine.
I wonder what their facilities are like?

<http://www.nyistatus.com/>

~~~
hga
Particularly important is that they're just in Zone C, i.e. apocalyptic
flooding required. They also kept hefty onsite fuel reserves, i.e. at one
point a reported 30 hours before they needed their first delivery, then 5
days....

This site, the/a main one for the Huffington Post (Datagram), they're all in
Zone A, when Zone B flooding was considered to be likely :-(
[http://project.wnyc.org/news-maps/hurricane-
zones/hurricane-...](http://project.wnyc.org/news-maps/hurricane-
zones/hurricane-zones.html) ). I'm sure Manhattan Island datacenter space in
Zones A and B cost less, but....

(I too use them for email and their siting has always been one of my biggest
concerns.)

------
jerrya
Why are the backup generators on the 17th floor and not the 3rd floor?
Assuming there is a very good reason for that,

Why wasn't there an additional pumping room on the 3rd floor, pre-built, with
a legal amount of diesel in reserve, and a additional pumps to take over from
the basement pumps when those fail, thus saving your bucket brigade 14 floors
of climbing?

Why are you carrying diesel in the open in 5 gallon buckets and not in fuel
containers that were purchased years ago?

All in all seems somewhat half-assed.

~~~
jmillikin
Generators are placed on the top floor to simplify the exhaust path, which
must terminate at the roof.

Pumps are placed next to the fuel because pumping liquid over any significant
vertical distance requires the pump to "push" rather than "pull". The fuel is
placed in the basement because nobody wants to sit next to a tank full of
diesel.

~~~
jerrya
"Generators are placed on the top floor to simplify the exhaust path, which
must terminate at the roof."

Thank you.

"Pumps are placed next to the fuel because pumping liquid over any significant
vertical distance requires the pump to "push" rather than "pull". The fuel is
placed in the basement because nobody wants to sit next to a tank full of
diesel."

The proper design would seem to have two pumping stages. One from basement to
3rd floor, the other from 3rd floor to 17th floor.

If fire codes are such that one can't safely store 24 - 72 hours of fuel above
flood level, don't advertise that your data center has reliable emergency
power backup.

------
mkr-hn
Can you use the elevators? If the generator doesn't have the extra power to
run them, offer some customers credit and a mention in the post-mortem if
they'll let you shut them down temporarily to power the elevators. Then you
can bring fuel up in drums instead of buckets.

edit: Nope. <http://news.ycombinator.com/item?id=4723814>

:(

~~~
tghw
The elevator equipment is also in the basement, which is flooded with sea
water and diesel fuel. Even if they had power, there's no running them until
everything is cleaned up.[1]

[1] <http://news.ycombinator.com/item?id=4720894>

~~~
mkr-hn
How much does it cost to rent a helicopter in NYC?

------
mkr-hn
I'm surprised to see so many building experts on HN.

------
Pyrodogg
With all of this effort put into keeping the data center running, I've been
wondering about a few things.

Was it actually connected to the outside world throughout the storm?

I have a hard time imaging that with the power out in large sections of the
city some key router on the line wouldn't have also lost power.

If that's the case, the effort was put in just to keep the computers warm to
prevent unplanned shutdown, not to actually provide uninterrupted service to
the customer?

I'm not familiar with data center operation. If you're already cut off from
the larger network at what point does it make sense to keep the machines
running vs. shutting them down?

Or perhaps i'm just mistaken and they were actually connected throughout. In
which case I find it amazing that the water knocked out pumps and necessitated
other shutdowns but their network wasn't damaged in some way.

~~~
madkangas
Yes, it's still connected to the outside world, and has been continuously thus
far. Example of a site still being served from machines at Peer1:
<http://blog.squarespace.com/>

(I am a Squarespace employee)

------
gallerytungsten
This story reminds me of the efforts one guy in New Orleans went through to
keep his data center running after Katrina. There were some tales of diesel
hauling in that blog as well.

<http://interdictor.livejournal.com/57475.html>
<http://interdictor.livejournal.com/40720.html>

<http://en.wikipedia.org/wiki/Interdictor_%28blog%29>

------
tlb
Some numbers: a 1 megawatt generator burns 70 gallons / hour. If someone can
carry 10 gallons (60 lbs), they need to make 7 trips / hour up 17 stories. I
think one soldier could manage it.

~~~
FireBeyond
Highly optimistic. Some numbers:

Columbia Tower in Seattle, Firefighter Stairclimb event (I think you could
agree a firefighter is probably on par with a soldier for fitness) - 63
stories carrying 50ish pounds of gear, average finish time, 48 minutes.

7 trips an hour up and down 17 stories = 119 stories.

Oh, and the firefighters are exhausted, drenched in sweat, require cooling
down and up to an hour in rehab for each climb, with legs near collapse,
burning like fire...

If I had to do this, I'd be going with the bucket brigade, every time (spoken
as someone who has completed that stairclimb event).

------
ryan_s
Was it worth it? Really?

~~~
Moto7451
I don't think so. At work we set up some servers on the west coast to take
over in case our main provider on the east coast went down.

I'd say the chance of one of their people getting hurt isn't really worth
anyone's uptime.

Also all the single site prep in the world doesn't help if that one site is
taken out completely. Keeping multiple servers in multiple areas is a must if
100% uptime, even during events like this, is key.

~~~
thomblake
If you know of some way to swap out servers with 100% uptime, I'd like to hear
it. Even Stack Exchange, which had that sort of plan in place, had to go
'static' for about a half hour.

------
asher_
Awesome dedication, but it makes me wonder why geo-redundancy isn't in place
for companies the size of these.

------
qq66
The long-term correct solution to this problem is cloud infrastructure with
multi-provider failover. If you have a server in California hosted by Amazon
and a server in Texas hosted by Rackspace it's unlikely that you'll find
yourself hauling diesel fuel up a staircase.

------
nicholassmith
This got mentioned on an irc channel a few hours ago and my response was: "Why
don't they build a pulley? They're nerds, they have the skills". Obviously
shifting diesel about has some risks involved but a basic pulley system would
help save a lot of time.

~~~
eropple
Only if they have a place to rig it and the tools to do it. Neither are
guaranteed (and if they don't, they won't be getting them quickly).

~~~
nicholassmith
Depends how advanced and fancy they wanted to get it, there's probably enough
bits of random hardware in a data centre to improvise something.

It'd be more a 'thereifixedit.com' solution than something you'd use on a
daily basis, but it could (possibly) work.

~~~
tedunangst
Who keeps 17 stories of rope in a data center? Do you know how much 17 stories
of rope weighs?

~~~
nicholassmith
A data centre with a really comprehensive disaster recovery plan.

Don't necessarily need 17 stories depending on internal layout.

------
mindslight
Any bets as to whether they will still be singing that common 'implementation
efficiency doesn't matter, you can always scale horizontally' tune afterwards?

------
activepeanut
Is that dangerous?

~~~
pyre
I imagine that spilling diesel down the stairs would be hazardous. At the very
least, it's extremely viscous (slippery). In addition, that's probably one of
the fire escape routes.

~~~
dredmorbius
Appropriate username is appropriate.

------
smackfu
Is there an advantage of 17th floor vs. say 3rd?

~~~
grayrest
That's where the generator is.

~~~
smackfu
Ok. Is there some advantage to putting the generator on the 17th floor vs. the
3rd?

~~~
fr0sty
The building is 35 stories tall. the 17th and 18th floors are right in the
middle.

Besides that, I don't think the powers that be want people belching diesel
exhaust just a few feet above street level.

------
jaipilot747
What dedication!

------
7beersonthewall
Amazing!!!

