Hacker News new | past | comments | ask | show | jobs | submit login

maybe the disk filled up?

That would cause any disk-write-dependent service to crash, yeah, but the kernel isn't one of those. Syslog will stop and a bunch of things will complain, but I've always been able to SSH in and free up some disk.

Did you read/listen to the story? A non-tech person called him and said "the server crashed." I am pretty sure they would say that in case of "any disk-write-dependent-services crashed, but you could ssh in and free up some disk." The criminal boss wasn't gonna "ssh in and free up some disk", that's why he had an IT guy!

Anyway none of this is particularly important for the story, but I don't think the guy telling it is lying for dramatic effect, I think he's probably being honest that the boss saying "the server crashed" made him suspicious cause that server never crashed, and I too found it amusing for this to be included in the story (as a sort of by-the-by advertisement for linux). (Also, though: it turned out the server really did "crash" in some way).

I just figured he was talking about an OS-level "crash", considering how he went on about how Linux never crashes.

Non-IT people may say it's a crash no matter what actually happen. "Crashed" "Freezed" "app not working" "can't print" is the same thing for them often.

BTW I've seen an error where app server couldn't write files to a directory but only for specific filenames (that wasn't already created). Turns out if you have dozen thousands of files in one directory the hash table has collisions and some files you can create while some other names you cannot. It was lot of fun to discover that :)

And customers described it as "server doesn't work" but when we connected the randomly generated names it was trying to write were different and it worked.

This has happened to me on a Sunday albeit with a hosted server running that awful cPanel version of linux.

The backup was what killed it, it ran out of disk space and the box keeled over. I could not believe the backup program was that stupid to back up twice as much stuff as it had space for and then to kill off the important processes to keep the backup running until zero bytes were left.

I have also had a close one with mySQL replication, it took the disk to fill up before I configured it to purge the logs. My own stupidity is to blame for that one.

Log files are going to be the killer, run a linux box for long enough without any log file rotation and the disk is ultimately going to fill up. I can't imagine that a decade ago when this server was built that there was a rack of terabyte SSDs in there.

Email is also an area that just grows and grows. The email doesn't even have to be used, just your system message stuff.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact