

A human error caused some sensitive server... - GVRV
http://staff.tumblr.com/post/3959106211/update-regarding-security-issue

======
elliottcarlson
Since all that really can be deferred from the dumped code is that someone
made a typo in a globally included config file (whether it was outside of
webroot or not can't really be determined). The obvious issue is that
obviously there was no process for testing code prior to deployment. This
could have been caught using a standard Dev/Stage/Production environment
setup, or even a quick test prior to deployment to Production. Other possible
ways to catch this is with proper Unit Testing (which would have broken when
not a single service would work due to no config params coming through), or a
proper Continuous Integration environment to handle all that.

------
krobertson
I'm really beginning to wonder what the hell is going on at Tumblr. Constant
downtime to where their down message is the new fail whale, cases of severe
downtime with poor communication, and now this.

Hopefully the shape up. I'd have serious concerns even running a personal blog
on there.

------
j2d2j2d2
Aren't all bugs effectively a human error at some level? In this case, the
human wrote bad PHP.

I hope Tumblr will keep us more in the loop on this than when they were down
for 24+ hours.

~~~
knowtheory
no?

Bugs can arise when a system or code base is being used in a manner that was
not originally envisioned by its author.

That's not a bug, that's something more akin to failure due to scope creep.

~~~
j2d2j2d2
I fail to see how that isn't also a human error?

~~~
ejames
Yes, but it's a different and more serious kind of human error.

"Someone made a typo" is one kind of human error. It's common and easily
corrected. "We are running our servers such that someone can ruin everything
by making a typo, and we won't catch it until it's too late" is a different,
more serious and less easily-corrected error.

But yes, all bugs are human error.

~~~
Cococabasa
No, all bugs are not human error.

Some bugs are caused because the programmer cannot see into the future and
determine that something they programmed will be used in a manner that was not
intended (using a Hammer function instead of the new Screwdriver function).

So the bug is in the code that the programmer wrote years ago. It is not
suitable for today and WOULD crash and create a bug if used that way.

But is there an error somewhere? No, the programmer just cannot see into the
future.

~~~
ejames
Ah, but that is human error - the error of the person who used a program in a
manner for which it was not intended! They erred in not understanding that the
program was written years ago and not intended for that use. Of course, then
it is not the error of the human who wrote the program.

This, however, is the reason why the phrase "human error" is mainly applied to
simple, 'random' mistakes such as typos, fat-fingering a button, or misfiling
a paper - errors that anyone can make, simply by being a forgetful human -
rather than the literal meaning of "any error committed by a human". Stretched
to cover every mistake made by a homo sapiens, the phrase loses all
descriptive power.

So, to clarify - all errors made by humans are "human error" in some sense,
but that renders the classification meaningless, so we should distinguish
between simple, common errors and more worrisome ones by using words other
than "human error". Your example of a programmer that did not foresee a future
desired use could be referred to as a "requirements error" if the programmer
reasonably could have or should have foreseen it, as "operator error" if the
person using the program is meant to be trained to understand its proper uses,
or a "misallocation of resources error" if the problem exists because not
enough budget or time was assigned to keep the program aligned with current
necessary uses.

I sometimes feel that a fair number of disagreements or disputes in the
comments of Hacker News are best resolved by checking whether we are using
words in exactly the best way.

------
itsnotvalid
I think they need to further explain what _exactly_ happened than simply
stating "human error". If it is avoidable, admit what it is and also let other
people to learn this lesson.

What configuration error (technically, by definitions from some IEEE
guidelines, mistakes) did they make to have this happening?

~~~
donohoe
No. I don't think they do. They've made a mistake and as long as they are
willing to take responsibility for the consequences then I don't see the need
for a public post-mortem just to satisfy our curiosity.

~~~
itsnotvalid
I guess you're right. Anyway the reason is pretty clear enough after actually
reading the pasted code.

------
jrockway
_The fact that this occurred at all is still unacceptable, and we’ll be
seriously evaluating and adjusting our processes to ensure an error like this
can never happen again._

I'm not aware of any theorem provers for Apache config files.

------
nbpoole
See also; <http://news.ycombinator.com/item?id=2343330>

