Hacker Newsnew | comments | show | ask | jobs | submitlogin

The datacenter I use, has a policy to do full load runs for 30 minutes each month.

AWS would have found this, and been able to fix it in a timely fashion, if they did the same (the genset lasted for 10 minutes under load before failing).




Most of the generators I'm aware of require this for maintenance purposes anyway - without running them occasionally the lubricants will freeze up and sieze the engine.

-----


All sorts of stuff.

Diesel is great food. For bacteria, that is. So it's treated, but you've still got to stir it to keep gunk from settling, you've got to rotate it (so you burn through your stock every so often), you've got to filter it. And stages of all of that can go wrong.

I recall a cruise on a twin V12 turbodiesel powered ship (hey, we've got full redundancy!) in which both engines failed. Cause? Goopy fuel and clogged filters (she spent a lot of time in port). This happened a couple of hours into cruise, fortunately on inland waterways, not open seas, shallow enough water to anchor, and numerous skiffs with which we could head to shore and find replacement parts.

More recently, an colo site I know of was hit by a similar outage: utility power went out, generators kicked in, but a transfer switch failed to activate properly. Colo dumped.

Second time it was the fire detection system which inadvertantly activated. One of its activation modes is to deenergize the datacenter (if you've got a problem with too much energy liberation, not dumping more energy into the system can be a useful feature). Again, full colo dump. And APCs will only buy you enough time for a clean shutdown. If you've got them at all.

But, yes: exercise your backups.

-----


Yes, generators must be run priodically. However, not all data centers actually put the full load (or any load at all) on the generators during non-emergency periodic testing.

-----




Applications are open for YC Summer 2015

Guidelines | FAQ | Support | Lists | Bookmarklet | DMCA | Y Combinator | Apply | Contact

Search: