
The “Oh Shoot” Moment - bpyne
Seeing Adobe&#x27;s problem with a release erasing content for a significant group of users made me think about the developers&#x2F;engineers in the background. Did they have the &quot;oh shoot&quot; moment? I&#x27;ve had mine for much smaller issues. My heart raced, my mind raced with theories on what I missed, and I started to sweat.<p>What are your own &quot;Oh shoot&quot; moments? Did you suffer any consequences from management or anyone else in your organization?
======
BlameKaneda
We have a dev version of our intranet and a prod one. To make a change to a
page on the dev site, you drag and drop the HTML and JS files into a /dev-
SiteAssets folder.

One evening I wasn't paying attention and accidentally dropped the files to
the prod folder. I went to the prod page and sure enough, it was broken. It
wasn't a high traffic page, but I was still afraid that we'd get a complaint
or two.

Since the original prod files had been written over by the new ones, I went to
the page's GitHub and downloaded the files from the repo.

I then dragged the repo files to the prod folder, and the page was back to
normal. Definitely gave me a fright.

I informed my boss about what happened the next day and how it was resolved,
and although he looked somewhat alarmed I didn't get in trouble.

------
lexfiend
25 years ago, junior sysadmin me had to restore our server's drives from tape.
To my horror, all the backup tapes were corrupted. It took a while to figure
out what happened; my 2am cron job script basically did this:

    
    
      run_backup & eject_tape
    

Fortunately, we were able to reconstruct the server's contents from individual
workstations, but I had to endure "how's the backup?" ribbing for many months.

I also changed the script to:

    
    
      run_backup && test_backup && eject_tape
    

so if the tape was still in the drive when I got in to work, I knew to check
on things.

------
AnimalMuppet
I was bringing up a new CPU board with the OS on a hard drive. It contained
the only instance we had of the OS for that CPU board. And somehow I managed
to delete the file that contained the OS kernel.

The board still booted, because the boot sector still pointed to the sector
that the OS kernel file had been in, and the contents of those sectors hadn't
changed. But the sectors were marked as unused on the disk, and could have
been overwritten by any file write.

I booted for two days from that deleted file before I got to the point where I
could re-create it.

------
rococosbasilisk
Tom Scott has a great video on his "Oh Shoot" moment
([https://www.youtube.com/watch?v=X6NJkWbM1xk](https://www.youtube.com/watch?v=X6NJkWbM1xk)).

Thinking back on the times I've had code exist nowhere else but the
clipboard...

