
The tech world is rallying around a young developer who made a huge mistake - hourislate
https://qz.com/999495/the-tech-world-is-rallying-around-a-young-developer-who-made-a-huge-embarrassing-mistake/
======
ng12
This article clearly missed the point. He didn't make a huge mistake, he made
a very tiny one. The company/CTO made the huge mistakes.

~~~
protomyth
A old supervisor of mine told me that "You treat a new developer like a
duckling near a pond with Northern Pike". There is no way you let them have
access to production (or frankly, anything past a development environment)
until they have been there a while and gotten the hang of it.

On a tangent, I was paranoid enough in one position to have all my production
terminal windows have a red background with yellow text. That brought the
proper, serious attitude towards my work.

~~~
zaphod12
That's not paranoid! That's a fantastic thing to do and one that I highly
recommend for anyone. It's terribly easy to forget which environment you're
working in when you get distracted for a moment and that's how mistakes
happen.

~~~
protomyth
I went the whole nine yards on that project. I had white (black font) for my
own machine, green for development, blue for integration test, yellow for
system test, and red for production.

I liked the green / white text theme the most with blue being my second
favorite to write code in so those got the ones I would use the most. I was
never a lover of the old amber terminals and red is just plain painful to
write on. I guess the more pain your terminal causes you, the more dangerous
the environment.

~~~
wsinks
Was it automated? If so, how?

~~~
tbirrell
I have a similar setup, and it is relatively easy to set up using .bashrc and
terminal color variables.

~~~
throwanem
I did something similar with a shell script for MySQL databases, both to
support shorthand invocation (i.e. "mysql foo-bar-baz" to look up the named
configuration, expand it into whatever set of arguments is defined therein,
and invoke mysql(1) accordingly), and to allow for detailed and environment-
specific prompts which warn me when I'm targeting production.

It's hideous, of course. But it works very well for me, and I don't suppose
I'd be averse, if interest becomes evident, to sharing it somewhere for others
to benefit from and/or peruse in horrified fascination.

~~~
throwanem
It's more hideous than I remembered! But for anyone interested, here you are:

[https://gist.github.com/anonymous/8ec128e88dac85d93ed58fb518...](https://gist.github.com/anonymous/8ec128e88dac85d93ed58fb518b2c705)

Outside GNU coreutils, the only dependencies are mysql(1) (of course) and
jq(1), which is more or less "awk for JSON" and which I strongly recommend to
anyone who regularly deals with that file format.

Share and Enjoy!

------
pier25
I've said it before and I'll say it again.

It doesn't make any sense that anyone would put a junior on his first day even
close to a production environment, much less using a tool that can delete a
database that btw you can't restore from a backup.

IMO this is either a false story, or this was done on purpose.

What if the company wanted to get rid of some data that would put the company
in a difficult position in front of its shareholders or even the law? What if
the lost data could prove that the CTO had made some other much bigger mistake
that allowed hackers to get private information?

We will never know of course.

~~~
dlp211
Nah, this is pure Hanlon's Razor right here: "Never attribute to malice that
which is adequately explained by stupidity."

~~~
amenod
That still only applies to the second option. The first one (false story) is
very much possible and it seems very plausible to me. It could be a bet, a
sociological research or one of those "why not" ideas... we'll probably never
know.

------
throwaway7645
Did anyone ever confirm it wasn't a hoax? Or find the actual company?

~~~
neuromantik8086
I mean, wasn't the crux of this story a plot point in Silicon Valley? Either
its a hoax or Silicon Valley is truly uncannily realistic.

~~~
iaw
> Silicon Valley is truly uncannily realistic.

Silicon Valley and Veep are both disturbingly prescient.

------
cdubzzz
Fun story! A long time ago I interned for a very large catalog company. It was
my first real internship in IT.

One of the first things (so a couple days in, not exactly day one) the CTO had
me do was "map" their server room (e.g. figure out where data and power cables
were going to and coming from, physically and virtually). While working my way
through the back of a massive UPS rack, I stepped on one power cord that
happened to traverse over another 10 or 15 cords. I did it in such a way that
all of those cords were pulled ever so slightly from their UPS connections. I
didn't even realize it until one of the staff IT guys came in the room a few
minutes later and asked me what happened. Turns out those cords powered the
300+ Citrix thin clients in use by the national call center downstairs.

I thought for sure I was in huge trouble, but everything came back online
fine, the staff IT guys poked some fun at me and the CTO helped turn it in to
a valuable lesson. I learned a lot of great stuff during that interneship and
to this day, 15 or so years later, I still think of that experience when doing
risky work (or when I fuck it up).

~~~
djsumdog
Sounds like a cable fail:
[https://www.reddit.com/r/cablefail/](https://www.reddit.com/r/cablefail/)

------
godzillabrennus
Glad to see more folks discussing this. I'm sure it's reassuring to the new
developer that this wasn't their fault.

It's still amazing how badly that company is being managed from a technical
perspective.

~~~
runin2k1
Or... since no aspect of the story is verifiable, and much of it is
nonsensical in the reactions of all parties involved, it is completely fake.

~~~
steveb0x
What would be the motivation behind fabricating a story like this?

I see posts of this nature all the time on r/csquestions. This one just
happened to go viral.

~~~
runin2k1
Attention seeking behavior. Maybe a clever teacher asking students to create
fake news to see how easy people are to fool. Boredom.

See -- Programmer Automates His Job Away for 6 years and then forgets how to
program.

I don't think any individual aspect of the story is unbelievable, but when
taken as a whole it stretches credulity.

------
spapas82
The original story (junior dev was given access to production / cleared the
database while trying to setup his dev env) was funny to me. Actually, I can't
really believe that what was described is possible. For me, there are two
explanations:

\- It's a prank by the OP. He posted a story out of his mind for whatever
reason.

\- It was all manipulated by a different employee of the company. He (by
accident) broke the production db and decided to blame the junior dev: Most
junior devs will just believe you (out of fear and inexperience) when you say
to them that they broke the production.

The 2nd one seems more possible to me.

------
heisenbit
Once I was put in charge of running regularly some DB scripts for a social
site (non profit, not a payed job) with about 10K active users. It was all
manual as the nobody had the time and skill to automate it. There was no
development site, there was only production so adding any automation was also
risky. And of course one fateful Sunday I pasted a script that was not
properly adjusted and the DB was fubar. It took us more than a day to recover
- thankfully we had a recently current backup. Later we all spoke about it and
also did a write-up. The debriefing helped a lot processing the shock but for
a long time I loathed to log into the DB.

When I'm fully in control I engineer things in a way that make disasters
unlikely and easily recoverable. But that is not always possible. I tend to
avoid touching any production DB but again that is not always possible.

I always believed I have a developer mindset more than an operational mindset
and worked more in roles compatible with the former than the larger. When
looking at the devops culture I always wonder whether the lessons of strict
control of production environments are getting enough attention. There is
value is closing the gap between the two world but please leave a gap.

------
neoCrimeLabs
Unrelated to the story, but I've had a similar experience as to some of the
stories described in the article - where a cascade of unseen problems landed
on my lap and I took the blame even though little of it was my doing.

(Note, this was many years ago and mildly traumatic, so some details might be
fuzzy)

In 2001 I was working for one of the larger internet server co-location
companies in the US as a dedicated customer support engineer (remote hands)
for several large clients.

One of my clients requested that I perform a memory upgrade on their Sun 420R
servers at 4am (on my birthday none the less).

So I come in. Confirm with the client they are ready to be shut down, confirm
with the NOC that the servers are coming down. The client remotely shut down
the servers in question but they didn't power off (missed clue). One by one I
power them down, roll them out of the rack, insert memory.

This was a pretty routine request, so it went smoothly - so I had thought.

Switch the console to the database server, boot. The kernel starts without
issue, then a slew of error messages. Too many to read.

The Sun 420R's were known to have memory riser issues, so I power off reseat
the riser, and all the memory. Same problem.

I take out the new memory. Same problem.

Before I could troubleshoot the client guesses it's a hardware issue, suggests
we bring up the other systems and the cold-backup systems to compensate.

Nothing comes up.

Not

One

System

So at this point the client starts to panic over the phone. They hang up on
me.

So I pull out my laptop and switch to a serial console on one of the servers
in attempt to troubleshoot the error messages.

All the error messages are Bourne Shell (sh, not bash) errors.

Realizing this doesn't appear to be a hardware issue, I boot from a recovery
cdrom, and start inspecting the init system. Everything looked great. So I
tried to run a script, and then it became obvious - everything LOOKED too
great.

All the init files had been reformatted. It said as much in a comment. A
certain very well known perl developer (at the time) who developed the site
for the client had found the formatting of the init scripts atrocious and
unreadable, so took it upon himself to reformat every init script on every
server. The problem is, bourne shell didn't like some of the formatting
changes - mainly space between variable, equals sign, and value declarations.
Had he tested his changes, it would have been obvious what the problem was.

Testing my theory I fixed the formatting of a single script, and it
immediately works without error. Problem found, now I need to fix all the
scripts on all servers.

I call the client. No answer.

I call the client again. Answer. "Did you fix our servers yet?" "No, I
found..." phone disconnects.

I call again, no answer.

I call again, no answer.

I call the NOC, asking they try to inform the client of the status.

So I proceed to start fixing the first server. It took an hour, but the server
came up. The second server came up a little faster, and I proceeded to work on
the third and fourth.

Meanwhile, I find out the client had been calling around escalating to senior
management at my employer petitioning that they fire me and put someone
competent on the job.

I find this out as one of the analysts comes up from the NOC and tells me I'm
relieved, and was asked to go home with no further information. So I explain
what had happened and what I was doing to fix the issue.

The next day I am called into the office. Told I was fired for my obvious
screwups, and that thankfully the NOC analyst had been there to save them from
losing such a large client.

Happy birthday.

Without question, there were things I could have done better in that
situation, but could not have predicted in a million years that someone would
reformat init files without testing them.

~~~
illumin8
Wow, incredible story. I used to be a Sun SSE, so I can feel for you. I know
hindsight is 20/20, but this was my approach when replacing failed hardware
like memory DIMMs remotely (when the client wasn't there in the datacenter):
\- I always asked them to do a full shutdown,"init 5", and would wait until
the server physically powered off before working. That would verify I was on
the right server. If the server didn't power off because the customer did
something wrong like "init 0" instead of "init 5", I would connect to the
serial console and verify that it was sitting at an openboot prompt. \- When I
ran into a software-related issue like "DB won't start," I would never touch
anything or try to fix it. I know you had great intentions, but fixing the
init scripts opens you up to a ton of potential hurt. As they say: "no good
deed goes unpunished..."

I really don't want you to think I'm criticizing your actions - you were a
true hero to the customer in a crazy situation. I think its truly unfortunate
that they terminated you for this. Best of luck!

~~~
neoCrimeLabs
Totally agree. I was much younger then, and learned a lot from the experience.
Even got a better job as a result. :-)

~~~
scrumper
Good for you taking something positive from the experience, but man that must
have hurt at the time! I know that at I at least was a lot less humble at the
start of my career than I am now.

------
eddyg
Previous discussion (718 points, 262 comments):

[https://news.ycombinator.com/item?id=14476421](https://news.ycombinator.com/item?id=14476421)

------
SubuSS
I am surprised they even have their prod data cloned into test servers: Get to
any reasonable size and this becomes a huge no-no. There were major measures
in MS/Amazon (where I spent a bunch of time) and now at snap to ensure prod
data isn't even visible without a bunch of hoops to jump through.

------
jzl
I'm going to quietly leave this obligatory Arrested Development reference
right here ...

[https://www.youtube.com/watch?v=GwQW3KW3DCc](https://www.youtube.com/watch?v=GwQW3KW3DCc)

------
jwildeboer
Did anyone offer him/her a new job? Or pro bono legal help?

