Hacker News new | comments | show | ask | jobs | submit login

You schedule someone (or hire a service) to provide 24/7 coverage.

If someone is required to respond outside of normal business hours, compensating PTO is provided.

Actually, I'd be strongly partial to that time off being granted at a multiple scale, much as overtime and double-overtime are rated. If I'm called on to respond outside of core hours (and am not normally scheduled for those hours), I get 1.5x time compensation. If it's between midnight and 8am, weekends, or holidays, it's 2x.

From Frank Herbert's Dune, there's the Fremen desert water compensation principle of 10x price -- the idea being that if you create an obligation of others to share their very limited water through loss, stillsuit damage, etc., you must pay them back 10 times what they provided you.

See generally: http://dune.wikia.com/wiki/Stillsuit

I'm also very much increasingly a fan of having engineering, not just ops, included on pager rotation:

• It increases pager coverage, and reduces any one person's pager obligations. Simply having pager anticipation is a mental burden after a while.

• It creates a stronger incentive for response procedures: what are the expected obligations of response staff, what's considered sufficient effort, what's the escalation policy, who is expected to participate, what are consequences of failure to respond?

• It makes engineering more aware of the consequences of their actions: is insufficient defensive engineering causing outages (say, unlimited remote access to expensive operations), are alerts, notification mails, and/or monitoring/logging obscuring rather than revealing anomalous conditions? Are mechanisms for adjusting, repairing, updating, and/or restarting systems complex and/or failure prone themselves?

My experience at one site, where I was a recent staff member (and hence unfamiliar with policies, procedures, and capabilities), systems went down starting at 2am, I was unable to raise engineering or my manager, and the response the next staff meeting to my observation of this was pretty much "so what" did not endear me to the organization (I left it shortly afterward).

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact