
Ask HN: What's your company on-call setup? - anoncall
I&#x27;m a junior programmer at a mid-sized company (150 employees) with an engineering team of 20 developers.<p>I&#x27;m curious what is typically expected in terms of responsibilities, timing and compensation.<p>- How often are folks on-call? How frequently do issues occur at your org?<p>- How much do people get paid? Is it extra when issues happen?<p>- Is it normal to have all developers on rotation? Or do you have a separate operations team?<p>- Can you work remotely? Are you expected to be available at short notice? Do you have a second level backup person?<p>Any other insights I should know about are welcome. Thank you! :)
======
iends
I work on a large SaaS product that is deployed in AWS globally and is
expected to have 100% uptime. Each service team in the SaaS has its own on
call rotation and is responsible for being highly available. There are also
architects and more DevOps folks (SRE team) always on call so you can always
get help at 2am if you really need it.

I'm the primary on call every 3 weeks, but secondary the week after.
Technically, since I'm the team lead, I'm always on call if the issue
escalates to me at level 3. In practice my team is very good and it's
extremely rare to get a page when you are not primary.

The team doesn't get paid extra, but it is mentioned in the interview
completely up front and my company's compensation is pretty solid.

If paged after hours you are expected to work remote. If you can't get to a
computer in 30 minutes you should escalate the issue (or just ignore the
page). This is company policy, but I expect most companies to have some type
of SLA for responding to pages.

As far as what type of pages we get...when there was flooding in AWS Australia
a few months back I got paged because we lost a certain number of AWS nodes.
Sometimes we deploy code with a memory leak and we find out a day or two
later. Often, AWS just does something stupid (like the AWS S3 outage a few
months back). Sometimes the DB cluster loses a node. Sometimes we ship code
that interacts with another service in unintended ways. We NEVER interact with
customers, all our pages are in response to specific technical issues. If
customers need to be contacted there are people with customer service skills
that convey the message appropriately. We do not get paged out for issues that
are not major (e.g. the product or a major feature stops working).

------
odonnellryan
What's expected? As an owner of a software company that requires on-call staff
(I'm basically 100% on call all the time): If they did not mention it during
the interview, nothing is expected of you on-call. If they bring that up,
renegotiate your contract.

\- How often are folks on-call? How frequently do issues occur at your org?

I worked in IT that had a rotation. We'd have 6-8 techs and just rotate
weekly. However, issues were infrequent: maybe 1 every 2 weeks. Minor things.

\- How much do people get paid? Is it extra when issues happen?

Depends. Maybe your total compensation is worth it to you. Maybe not.

\- Is it normal to have all developers on rotation? Or do you have a separate
operations team?

It's absolutely normal to have only some devs on rotation. It depends on your
business.

\- Can you work remotely? Are you expected to be available at short notice? Do
you have a second level backup person?

Legally, at least in most US states (if not all), if you need to be available
you must be paid for it if you're not salary exempt. So if you're not being
paid, feel free to go camping.

Always have a second level guy. Then a number that gets called to ping the
CTO/CEO if that second level guy doesn't check in. There are systems for this.

------
klez
All of this depends on the type of service the company offers, your contract,
your country, and other stuff.

For example in my company we have 5 support engineers who self-organized their
rotation. The important thing is that someone is available from 1 pm to 2 pm
to 6 pm till 8 pm, because one of our clients pays for that time range.

The engineer needs to answer phone calls and emails in a timely manner,
depending on the issue (from 30 minutes for issues that impact business
operation to 4 hours for minor stuff, like resetting a password).

They get paid a little for the time they're available and roughly the same as
overtime if they actully need do something.

This is in Italy and, I want to really stress this, it's how it works at my
company and with our customers.

------
wayn3
I truly don't get what kind of company is expecting software engineers to be
on call.

What do they even want you to do about issues clients have with software? If
the software is down for some reason, call devops. Or call AWS. I don't care.

If there's a bug, its not going to get fixed at 4am by one person pushing a
hotfix through production. Although if thats how bugs get fixed at your
company, maybe you should revisit whether thats a good approach to devops and
when you find that its bs, you can finally abolish "on-call rotation".

"btw, you're on-call.." "if I wanted to be on call, I'd be in medicine."

~~~
odonnellryan
> I truly don't get what kind of company is expecting software engineers to be
> on call.

What if you have clients who pay you big $$$ and they work all the time?
That's not that unusual.

> What do they even want you to do about issues clients have with software? If
> the software is down for some reason, call devops. Or call AWS. I don't
> care.

Very narrow view. You can have a software team of 4 people and bring in 10-100
mil contracts. There's not a DevOps staff in this case. And what would AWS do?

> If there's a bug, its not going to get fixed at 4am by one person pushing a
> hotfix through production.

I've solved 100s of problems ~4am by doing just this.

> Although if thats how bugs get fixed at your company, maybe you should
> revisit whether thats a good approach to devops and when you find that its
> bs

\- Buy/start to manage legacy app for XYZ company

\- Legacy app breaks

\- Need to fix it

None of that has anything to do with the software company being a bad company.
Things break.

> "btw, you're on-call.." "if I wanted to be on call, I'd be in medicine."

I'd be super mad if I didn't know I needed to be on call at a company and they
just told me I needed to be.

I'm not arguing that it should be expected in software, just that there are
plenty of software companies that require on-call staff, like mine.

