
Ask HN: Your Biggest mistake at work - lelima
The other day, I almost drop a production table, with many users, in a bank.<p>Thank god I checked one last time the environment variables before running into a disaster.<p>Have you been in a huge mistake involved? what happen?
======
lucozade
This was a long time ago.

I'd just taken over as "team lead" of the internal support desk for an
interest rate swap trading system at a tier 1 bank. By team lead, I mean that
I was nominally in charge of the one contractor who was vastly more
experienced than me. She was also famously prickly, particularly with upstart
novices such as myself. The lead role was considered a right of passage
because of this.

Anyway, one morning she comes to me, ashen faced. She had just deleted the
first leg of every single swap that the bank held. At that time, this would
have been a relatively high proportion of every swap traded in history. We did
have overnight backups but these took about 8 hours to bring online and the
prod DB would need to be down for all that time. Such was Sybase back in the
day.

Now, it just so happened that one of the operations team leads was a drinking
buddy of mine. I called her and asked if there was any way that they could
stay out of the system _for the whole day_. Bless her, she re-arranged
schedules, got people to come back in later, pulled in favours from the US ops
team etc etc.

In the end we managed the recovery and the day's trading was overlaid without
any management escalation happening. Something close to a miracle.

Suffice to say that, from that day on, said contractor was the epitome of
sweetness and light to both me and the ops team.

------
bryanrasmussen
Huge mistake I caught a consultant making at the same company where I made the
Sinatra server mistake.

The company sent email alerts to customers whenever data changed. It turned
out that a lot of people were not getting emails and they hadn't been for some
months. The alerting system was written in Java (not my language), and was
written by a part of the company in Argentina so not really anything to do
with us.

The consultant was tasked with finding the bug, he stayed home to work on it
so he could have no distractions.

I thought well I will look at code changes for the last few months, maybe I
will notice something - and it turned out that the consultant tasked with
finding the problem had some months ago made a change to the Java code dealing
with alerting, which was weird. I went and looked at the before and after -
the before was typical verbose Java code and I could sort of follow along and
see it was for sending off alerts whenever the data changed ok. The new code
seemed to just drop off into nothing halfway through.

So we were having a phone call with the consultant and he said well I am going
to talk with 'head programmer in Argentina' in an hour to find out but can't
figure out what is wrong.

And I Was like well I was looking through the code...

and you could see the consultant being somewhat annoyed at my chiming in
because hey, what is Bryan doing looking at Java code he doesn't know crap
about that language. I have to humor him, the big doofus. ugh. politeness
sucks.

and I said so a could months ago somebody changed the code in this Java file
here and their name was 'consultant name' and maybe that guy could help us
figure out what the code is doing, because I don't think it will ever work.

And it turned out I was right and he was wrong and that was why many customers
had not gotten their alerts for a couple months.

To pay for it he had to bring everybody pastries the next day.

About a week later I shipped a bug that made a bold text not bold on some
types of documents. To pay for it I brought the biggest fanciest cake I could
find, but I don't think anybody understood why.

------
bryanrasmussen
huge mistakes I've made:

I made a little Sinatra server to get some data from an oracle database and
then put it into a Redis Queue to be analyzed by a bayesian classified -
80,000+ documents.

The first version of the code was a little repetitive so I put a main part
into a function, checked it worked, went home for the weekend.

Turns out after my code change it was opening a new DB connection for every
document. Someone else caught it and turned it off, wasn't that big a mistake
anyway because not in production but still embarrassing.

One time working international standardization, I asked one of the people on
the committee I was on a question, sent to the mail list of course, I got a
long 3 page email in response. I started reading, all they were doing was
trash talking one of the people on the committee, who I thought was also one
of the more helpful people to be honest. So after skimming through and
realizing they never answered my question I answered some nice little thank
you thing.

However I had my outlook configured to send stuff automatically to the email
list when it had to do with this stuff, and I didn't notice her thing at the
bottom of the email "for obvious reasons please don't send this to the mailing
list"

