Hacker News new | comments | show | ask | jobs | submit login

> These kinds of fuckups rarely have only one person at fault

Fat fingering an rm -rf or corrupting a DB, if you have good backup policies, should generally not cause End of Days.

Exactly. Accidentally deleting a database should be a minor concern in any well-run company. It'll be painful and annoying, but shouldn't cause "scrambling" for "3 days". The ire should be directed at management that allowed that condition to exist, not the person that fat-fingered something.

In this worldview "management" is expected to be superhuman, while the guy who actually makes the mistake is absolved of responsibility.

It's more of a question of demarcation of responsibilities, of which there are two:

1. Responsibility to have a backup.

2. Responsibility to not screw up live data.

When someone screws up on #2, it inconveniences the person responsible for #1 and potentially loses any data since the last backup. That is the limit of #2's responsibility in this.

If #1 hasn't done his job right, it will come out when someone eventually plays the part of #2 (mistakes happen). Once that happens, the damage from not having a backup is #1's responsibility, not #2's.

In this case, we do not have clear information about who was responsible for #1.

Speaking as a manager - no, not superhuman (I'll restate to be it's a management/organization problem - not an individual problem).

It is my job to make sure we have a well-run software engineering product team. Ensuring that backups run and are tested is one small part of my job. Not doing the backups, sure, but making sure the team has taken care of it.

Mistakes happen and folks shouldn't be punished for that.

Remember that disks are fallible - human error needn't be involved to require recovery from backups.

Regardless of the root cause of data corruption, requiring three days to recover is completely a management/organization problem.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact