Separately, Stack Exchange, which is in the same data center, is running off of its hot backup in Oregon.
We would have liked to have completely redundant data centers for FogBugz on Demand and Kiln on Demand to avoid even a few hours of downtime, but because those services rely heavily on giving every customer their own SQL database, there is almost no reasonable way to get fast failover to a different data center. We can do it with Stack Exchange because there are only a couple of hundred databases. We've been building a SAN solution which will make it possible to hot swap out to another datacenter, one day, but that project is not complete.
Why did you say you could do fast failover to LA in 2007?  It was the case then that every customer had their own SQL database too.
I'm not trying to be snarky, your team is obviously putting forth some heroic effort. I'm just an affected customer that is curious why the question seems to have two mutually exclusive answers.
In all fairness, Joel recognized in the article that the initial decision was based on their previous experience with Microsoft -- and AWS was still incipient in 2007 -- so it made sense at the time. I doubt this is still the case. Even less if you consider the costs of Sandy - duplicated servers, migration, hours lost, unhappy clients, etc.
And maybe this sounds snarky, but AWS is your "making it better" solution? Seriously?
NoVA didn't get hit that hard by Sandy, unlike NY/NJ. So data centers around here didn't tend to have problems, the few that I have equipment in didn't even loose utility power.
And I'm not even talking about Trello (which is still just a hobby for Joel & team), but this also brought down Fog Creek and all their commercial services and paying customers.
The positive side of Sandy (if there's any) is that people will really take more seriously the idea of "expect the best; plan for the worst". And Amazon will likely see a spike of new customers in the next following days.
My then-employer's building had its basement flooded; we were on floors 19 through 25, but our electricity and phones and Internet were gone --- and this, three weeks before the end of Q2. Our developers and IT people hauled computers down the stairs, and we moved to temporary space for several weeks, but we still missed, that is, we didn't achieve the sales and revenue targets that we had forecast for analysts.
I'm given to understand that because of the lessons learned from Allison, Houston isn't quite as vulnerable to flooding any more.
Another risk in New York that will take data centers offline, earthquake. It will happen at some point, but the chance of it happening in the next 100 years small (contrasted to Santa Clara where we're much more likely to get an earthquake than a flood). So you could pay to have your Manhattan high rise put onto base isolators and proofed against an earthquake, but what size earthquake? Magnitude 3? Magnitude 6? Magnitude 9? And then if your building is still standing brightly after the Magnitude 9 earthquake are your fiber optics still there? How about the network tie point? Did it fall into a hole in the ground? So the cost to make all of Manhattan resistant to a 9.0 earthquake?
Its impressive that they are carrying up the fuel. I might be inclined to see if I could tractor in a 12kW generator to run one elevator. Sure you'd be burning fuel at both ends but it would be easier on the crew hauling the petrol.
Hospitals should definitely be built to handle a magnitude 5 or more. You're talking about a high probability of loss of life if they can't handle a 1 in 100 year event. Data centers, who cares? It's cost vs expected damage, and unless it's a safety-critical server (emergency services co-ordination?) you're probably just as well off shipping the servers somewhere else and eating the cost of the downtime.
As for getting a 12kW generator in a natural disaster, generators are in extremely high demand. You'll have trouble finding one. The other fun thing about generators in a disaster - people usually don't test them, and end up panicing in a dark basement with no idea how to switch them on and hook them up.
Having been lucky enough not to be placed in the situation you are in, I doubt I could have gone to such lengths to get everything back online so quickly. I admire everything you guys are doing.
You might as well blame a data-center for going down when the building it's in burns to the ground.
An always-submerged in-tank pump, not unlike that of a water well, powered via a sealed line from the generators above could, in principle, avoid this problem.
I hope the generators are able to power some sort of mechanical lifting arrangement. Manually hauling the drums is mighty hard.
Still, the extra height isn't going to buy you much.
According to omniscient Wikipedia, diesel is 83% the density of water, so the 33 feet that water can be lifted via suction becomes ~38'.
The Trieste used the buoyancy of (incompressible) gasoline to enable a return trip from the bottom of the Mariana Trench.
Strong work lifting all that fuel!
I've never done the full 15 year assessment, but, on average, the smart people throwing themselves into hosting my business applications have performed at, or above, any level that I ever could.
People always talk about down time from third parties, but always seem to forget how much downtime self-hosting induces - particularly as you usually can't afford the 24x7 well-staffed NOC and operations team.
The temptation is always to say "I can do it better myself" - but the reality is you probably can find someone out there who can do it better than you, and, even when you "Self Host" - 95% of the time that really means relying on someone else to provide you a data center with generators/diesel/hvac/security/Sonet/etc...
With that said - Fog Creek is a tiny little operation, that probably operates at, or below the level of our own Operations team - so it's a coin toss as to whether they would do better than us at hosting.
On the flip side - I'm pretty confident that gmail, on average, has had better uptime the last 10 years than the mail servers I've managed. (Though, the last 5 years have been remarkably stable for self hosted email. Exchange has come a long way...)
Given that this is a unpopular view around here anyway, I want to add another thought:
If your company has it's own webpage at datacenter A, including payment from datacenter/service B with CDN C and Twitter-integration D, using hosted FogBugz for support mails, hosting it's CSS-stylesheets on TRELLO which was baked by Google AppEngine, and using email provider E...
Why don't you have downtime at leas one a quarter???
How can a system that relies on so many components and other companies be reliable? With all those AWS-outtakes?
I don' understand this.
If you host most of your stuff yourself, either everything works, or nothing. But not all goes down at the same time.
However (regarding AppEngine based solutions), as far as I know, if AppEngine goes down, there is no way to have any backup, is it? Only google can host AppEngine apps.
It turns out that many backup power plans have not been designed with longterm flooding in mind.
Why aren't data centers built where natural gas is available? A natural gas powered generator could run from the gas lines, which should be more reliable than fuel that has to be trucked in.
Keep in mind that Manhattan has NEVER flooded before.
I can't for the life of me think why someone would put a backup generator 17 floors up, these guys don't know you can buy mains cables longer than 4 feet?
If it absolutely has to be 17 floors up, buy a fucking pump.
Geeks. Sometimes we're so dumb it hurts.
"Here's the physical situation:
The generators are on a high floor in the building and the pumps supplying the generators with fuel are submerged. The best option at this point is for people to physically lug diesel up over a dozen floors, or make other arrangements for pumping fuel to that high floor. "
Yes, a generator not in the basement / ground level makes sense. When a flood happens, it usually happens from the ground floor up ;) Thus why generators are at the upper levels.
Even in this case, using a pump solution makes little sense. Assuming you have electricity to power the pump, I suppose you could power the pump with the electricity from the remaining fuel, but a 17 floor pipe containing fuel would probably be heavier (and more dangerous) than a long mains power cord, and it wouldn't make sense to have a 17 floor cable to power the pump with the pump at the bottom of the run than it would to have just the pipe of fuel running the length of 17 floors and the pump at the top of the run.
Time for a winch? I don't think Home Depot stocks a 17 story pipe, and relocating the generator probably will take more time than just winching / carrying up the barrels.
I understand that this might be a fire-safety issue. In which case I would agree that a winch should have been installed.
(I'm possibly missing something else obvious here!)
I suspect this is good advice, but the image that popped into my head was an elevated generator, a ground-level tank, and an electric pump reliant on either mains or generator power. "It's always worked during our monthly tests!" Whoops!
Probably the elevated generator should be installed with at least a small gravity-fed tank, easily refilled once the pump is operating. I doubt anyone wants a giant fuel depot anywhere inside or on the roof of the building.
Keep in mind that for heights taller than 3 floors (and practically, the limit is shorter than that), pumps have to be located on the same level as the fuel supply because the atmospheric column only has so much mass. Pumps and their electrical inputs should probably be installed with waterproofing if flooding is considered likely.
This has nothing to do with buying a pump, that was already there, this has to do with the fact that the diesel tanks and the pumps for them are under water.
In the future I hope they put the pumps on higher floors, and make sure the fuel tanks are completely sealed, so even if the tanks are flooded the pump can keep pumping fuel out of it.
You can armchair quarterback this all you want, but unless you're actually making the decisions, perhaps you should consider the information available.
Doesn't make that pump any less sensible though!
Past my bedtime, so didn't connect this with that wee bit of weather the East coast USA experienced recently.
Sincerely hope lugging diesel remains the least of their worries.