
Ask HN: How often are you on call? - buttman2077
Please share a few words on what size company you work for, and what size team you’re apart of. If you’d like to share, mention how often you’re paged or activated!
======
perl4ever
I just started working for the government, and I got yelled at for sending an
email on Saturday. The union contract says we're paid for overtime if we work
more than 40 hours, but in turn, we _never_ are supposed to work overtime
unless maybe the world is ending.

I used to work for a company that had a departmental phone and we would rotate
to a different person each week to be on call. However, waking people was
essentially eliminated by adding people on third shift (including me) and in
India.

In both cases, I was on a team of about 5-10 in an agency or division of a
couple or three hundred, within a much larger organization.

------
Proziam
I'm on call 24/7\. It's fairly normal to wake up at 2 am because there's a
pressing situation and a client is in need. That said, I actively chose this
situation for myself and find it to be pretty rewarding. Of course, it
requires some planning and coordination, so my team divvied up our
responsibilities accordingly to ensure that there are always resources
available for taking meetings on a normal schedule.

It's probably worth noting, we work with large brands and marketing agencies
from all around the world and are often called upon to 'save the day' once
someone's plans have fallen through. As a result, time is usually the one
thing our clients don't have enough of and being able to solve their problems
ASAP is what allows our clients to get back to normal operation. Given that,
it's easy for me to pick up the phone at 2 am because I know that whoever is
calling me is going to be having a worse time than I am.

------
caiobegotti
Small startup, tech team of about 10 (2 SREs + 7 devs + 1 biz, total company
size is 20, fully remote distributed across 3 continents), roughly 8 hours
twice a week and only during my local timezone business hours, then the same 8
hours in one weekend day every two weeks, also roughly. We have lots of
automated checks and alerts and only react to critical ones paging us via
Slack (most of the alerts that page us are known, some are flaky some are
false positives we can't ignore).

It was kind of forced on everybody (prematurely IMHO) but respecting people's
timezones and business hours was key to make it acceptable and you can take
the next day off no-questions-asked if you were on call on a weekend day even
if you didn't have to act on anything. It's far from ideal considering how we
still react to the critical alerts and what in them usually triggers actual
on-call support but I've had worse in the past so it's alright... for now.

------
geofft
Largeish company (engineering staff around 1000, I think?), and my team owns
the OS platform (packaging, config management), hardware purchasing, and the
installer, as well as our private cloud, which hosts a good chunk of services
(notably including dev machines, notably excluding the most money-making parts
of the business). A separate team is first-line sysadmin+SRE for most of the
company, so we don't respond unless you can't install machines or if _every_
machine is broken, or if the private cloud (as infrastructure, not a specific
VM on it) is acting up.

Private cloud was originally a separate team, and we had about 5 people on a
weekly rotation, where you'd do support and 24/7 on-call for a week. Now that
we've merged, it's about a dozen people on the on-call rotation, though we
still have a subteam of about 5 for the weekly rotation of who's paying
attention to private cloud support tickets during working hours.

Thanks to some work that we did a bit after I joined, we got the off-hours
paging to be much less frequent - in particular we no longer page if _any_
hypervisor is down, only if either a whole availability zone is down, or a
hypervisor hosting personal dev machines is down _and_ it's business hours.
(Otherwise, higher-level services ought to have failover between multiple VMs,
and devs working odd hours can use backup machines.) Usually that second
condition gets noticed by the support person (in Jira) before it escalates to
the on-call person. So it's rare to have anyone actually paged, and it usually
means something unexpected is on fire. So, any member of the team is as good
as any other at figuring out what broke and whether it can wait until morning,
which is why we were able to throw all of the combined team into the rotation
without training them all in everything.

There are still improvements I want to do, but the expected number of pages
rounds to zero (it looks like we've had 6 off-hours pages this year) so it's
certainly manageable.

------
antoinealb
SRE at Google. We are oncall for 12h then the other site takes over for the
other 12h, so no night shifts. You do that for 7 days in a row once every 5
week-ish. Some team have slight variations on that.

We get paged around once per 12h on average but our service is fairly young.
More mature services tend to page less I think.

~~~
wikibob
What is the on-call compensation?

Last I knew it was 33.3% time credited for on call hours with 30 minute
response requirement and 66.6% time credited for on call hours with 5 minute
response time. And it could be taken as additional holiday or as cash.

------
6700417
Large company. Team of 12. Do very specialized work.

On call: Never and never have been in 25+ years across multiple large tech
companies.

Seriously considered looking into a job at Facebook where this would have been
required and it was definitely one of the considerations that made that
position very unappealing.

------
esotericn
Currently contracting for a ~10 person company.

In the past I've mostly worked at small companies and have never officially
been on call. I've fixed the odd out of hours issue here and there without
there being a proper arrangement.

The systems I work on (trading / betting software) are such that by the time
you're through testing it's very rare to hit a game breaking issue because
there's a very large incentive to not like, lose the bank.

Dependencies in the critical path are kept to a minimum, etc.

Usually problems are more 'soft', like an API being turned off or changed or
something.

------
acranox
It depends on how many people we have on the team! Right now it’s every three
weeks. A while ago it was every 6 weeks.

I used to get paged dozens of times a week. Now it’s either once a week, or a
few dozen times a week (usually this is a blip that can just be ignored
because something went wrong and then fixed itself). Typically there’s one
small actionable item every week I’m on call.

The company is 500 people, my team varies in size a lot as people quit and we
try and hire replacements. It’s been as small as 3 and as big as 12.

------
twunde
I work at a 100-person startup. We currently do 1 week secondary followed by 1
week primary. This comes up every 2.5 months. On-call is 8am-8pm with any
overnight pages going to execs. I'll not that we originally had people doing
daily shifts and it was a huge pain for those who were religiously observant.
It was much harder to get someone to cover a Saturday or Sunday shift since
nobody wants to cover those. Will send much better

~~~
xchaotic
do you have any staff in other timezones to get better night coverage?

------
arboroia
I’d also be interested if people could add if being on call is (a) voluntary,
or (b) required on their team.

Also, it’d be good to know if on call comes with additional compensation?

------
stephbu
My tier 3 team is on 1-week point-Dev rotations. Roughly 1wk in 9 on primary,
same again on secondary. They take about 2-3 calls per primary+secondary. I’m
on Tier 4 in mgmt roster, and on call every other week. Take escalations and
fall thrus every 3-4mths. Tier 4 also plays duty manager during events such as
product launches.

------
Karupan
I feel I’m in the sweet spot. Work for a company based across 5 time zones.
Our team of 12 has optional out of hours support and people have to volunteer.
But it is only for Sev 1 issues where the entire app/service is down.

Since it is paid, we end up taking turns for the extra money. In the year I’ve
worked there, haven’t been pinged even once.

------
Sevii
Work for FAANG, team size is 6-8~. I'm oncall 24/7 roughly one week out of 6.
Usually paged 1-2 times per shift except during 'peak' periods but we usually
subdivide the rotation for those.

My worst shift I was paged during off hours twice, but was only woken one of
those times.

------
codegeek
I run a bootstrapped SAAS business with total team size of 12. I am
technically always on call if shit hits the fan. Not in the traditional sense
though. If server goes down (has happened like 2 times in past 5 years), I get
a call no matter what time of the night it is.

------
dmitrygr
Never, and I don't consider any job that demands this. Learn what work-life
separation means.

~~~
geofft
That's clearly not a sustainable answer for the world - professions ranging
from doctors to priests to soldier to sailor to CEO (can't ignore a crisis if
it happens on Saturday morning) to lawmaker all require working the occasional
unscheduled hour. And that's not even counting jobs like pilot or convenience
store clerk or paramedic that require at least some people to work on what the
world considers holidays (even if it's scheduled as part of their 40 hours a
week).

Is there a meaning of "work-life separation" that encompasses these
professions too, or shall we say the world relies on some people not having
it?

~~~
lostdog
If someone is dying, go ahead and page me!

Otherwise, companies need to hire night shifts if they want someone covering
while the devs are asleep. If you want full uptime, then hire for it! Your
employees shouldn't give up their sleep and weekends for a measly 0.003% of
your company.

Also, I would strongly support extra pay for low-wage workers working
holidays, weekends, or night shifts.

~~~
drstewart
It's pretty naive to think you can just hire random people to step in and
support complex business systems part-time.

Sure, you can have a global ops team that can deal with basic system issues
like "this machine ran out of disk space", but not "we're seeing a spike in
500s in our payment system because something that got deployed earlier today
broke a workflow". And saying "well just roll back!" isn't a solution either.
Maybe you can't roll back. Maybe that won't fix it because data is now
corrupt. Maybe it wasn't even a deploy to that system that broke things, but
some feature flag in some distant system. No one except the team itself can
diagnose these issues appropriately.

------
mikelward
Large cross-site team. We do 12x4/12x3 rotations (Mon-Thu, Fri-Sun) so
everyone stays fresh.

I've heard others are trying one-day rotations to make scheduling holidays
fairer and more flexible for people who have commitments on certain days.

------
cm2012
I own a marketing agency. I check my email often just on case but I rarely
have to hop on my laptop after hours. I'll probably pen an after hours short
email a few times a week though.

------
vanniv
Presently, one week in five. Average shift has one overnight page and one
other out of hours page (with high variance).

I am compensated for the oncall, but not a meaningful amount.

------
theonemind
Technically 24/7, except for clustered stuff that hardly goes down. More than
a year since the last time; I think. Thousands of employees, team of 7 or so

------
gvurrdon
Large university, team of 8. All out-of-hours work is on a “best effort” basis
only, i.e. I’ll do what I can (which may be nothing) if emailed.

------
cptn_brittish
Company the size of about 10.

I am the only person on call 24/7.

------
pyr0hu
Small bootstrappes company working on SaaS. Im the lead and has 2 programmers
on the team, so Im the one usually on call, 24/7.

------
astura
Never ever.

Current company has 400 employees my team is around seven but we'll have to
grow very fast very shortly.

------
probinso
Company size: 2 programmers

only when the other guy is on vacation

------
beginrescueend
One week out of two, for a company size of 20.

------
sidcool
Quite a lot. 30% of the day on average.

------
markyang12
once every 5 weeks, FAANG

------
whatthesmack
tl;dr One week out of four.

I’m an SRE on a team of 4 (incl manager). The company is about 140 people.
Each of my team members participates in on-call (incl manager) and is on for a
week at a time. We’ve dialed things in such that we might get paged once or
twice during that week and usually the pages wind up being minor/non-issues.
It’s really not bad, in my opinion.

