
Former Hostgator employee arrested, charged with rooting 2,700 servers - benhomie
http://arstechnica.com/security/2013/04/former-employee-arrested-charged-with-rooting-2700-hostgator-servers/
======
ghshephard
This all seemed like a pretty run of the mill story about an insider violating
company trust, and then getting caught - until the final sentence: "Among
other things, a desktop monitoring system that took screenshots of employee
workstations in one-minute increments helped Hostgator officials quickly zero
in on Gisse."

Not something I'd want on my personal system, but it's exactly the sort of
thing that I think every NOC/Secure environment should have for post-mortem
assessments.

~~~
gav
A previous employeer used Spector 360[1] on the majority of workstations. It
would monitor everything including taking a screengrab every 5 seconds that
you could then watch later.

They'd sit down employees and playback fast-forwarded video showing how much
time was wasted on Facebook, personal email, shopping, etc. It's horribly
invasive but it meant everyone was too scared to use work computers for
personal things.

[1] <http://www.spector360.com/>

~~~
hfsktr
My old job used spector. I'm pretty sure everyone knew it but people would
still be on FB playing games when you walked past. I believe IT was the last
to get it and for about a week after they installed it on our machines it
didn't work because Microsoft Essentials disabled it as malware.

I don't think they ever looked at it unless they wanted to fire someone and
didn't want to pay unemployment and needed proof that they weren't doing their
job.

~~~
stoic
This is pretty much why they had this at HG

------
ultimoo
Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain
processes and port numbers from showing up. I wonder how may one go about
understanding if they are using a 'hacked' version of something as non-trivial
and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I
wouldn't give it a second though and trust the output of these commands.

~~~
saurik
While this isn't guaranteed (all tools, including the compiler, may be
patched), you can use checks and balances: verify /proc doesn't contain
phantom processes, compile your own copy of ps, try more-obscure tools like
top. If by "understand" you just mean "notice"... well, you don't, until one
day you accidentally stumble across one of the above and start digging.
(Maybe, for example, you install some kind of server monitoring tool, and when
you log in to the web portal it provides you see a process that you find very
suspicious; when you use ps, it doesn't show.) In my case, I've noticed this
kind of thing twice: once, when the tool was binary pacthed to death (and just
crashed), and once when the "patch" was "replace binary entirely", and the
replacement was older and did not support a command line argument I knew that
it should.

~~~
ultimoo
Nice. So both times that you noticed this, was it malice on someone's part due
to which the tools were patched? I have just never heard or encountered such a
situation and am frankly paranoid about something like this happening to one
of the tools I use.

~~~
saurik
I've been pwned with 0-days in various email servers: sendmail over a decade
ago, and exim4 more recently (still many years ago, though). The patched copy
of ssh on one of my boxes was then distributing passwords to someone, and
which then was used to gain access to another machine.

What I'm always paranoid about is that I work in a community of security
researchers that sit on and occasionally drop 0-days: I have very little trust
that much software is actually remotely "secure". Meanwhile, the only reason I
had noticed those other attacks is just how sloppy they were... a more
targeted-to-me run by a more careful attacker would have maybe never been
noticed.

It has drastically changed the way I think about security, FWIW; as one
example: I don't every store logs on a box being logged anymore. Instead, logs
are immediately transported to another machine whose _only_ purpose is to
accept and store logs (and so is listening for incoming log packets, OpenSSH,
and nothing else. The first thing anyone does is attempt to patch themselves
out of logs (one attack I noticed because wtmp was mysteriously damaged).

------
jamescun
> While his root access gave Gisse access to private data stored on a large
> number of customer websites, there's no evidence he used it, the Hostgator
> executive said.

I think the article is quick to jump to the conclusion that he was attempting
to be malicious with his actions however this could be a case of Hanlon's
razor.

His actions _could_ easily be attributed to a less-than-aware sysadmin
developing his own solution to get around often arduous security restrictions.
Stupid, yes. Malicious, no.

~~~
tracker1
If he hasn't accessed any of these systems since he was terminated, he could
state that it was for "emergency" access to remote systems upon other
compromises. Since most of these systems are likely headless, then remote
access is the only way to get in. A lot of remote exploits will nuke SSH, and
other access tools, so having a "backdoor" is often a good idea.

That said, it's still likely that this guy is just a douche with a bad
attitude, and deserves everything he has coming. Big difference between this,
and "stealing" a bunch of reports that were government funded, and open to any
and all users on the school network they were accessed from.

~~~
homosaur
I don't think this would work. Installing your own solution on customer facing
hardware? Might get you out of prosecution, but I think you'd have a nearly
impossible time explaining that.

------
dotmanish
Did these 2700 servers play a role in any DDoS attacks as well?

It would be quite a lucrative stance for the employee to sell access to these
servers to one or more groups who could potentially make more _use_ of them.

~~~
cperciva
2700 servers all on the same network makes for far less of a DDoS attack then
2700 similar servers on different networks - and it's far easier to detect and
block too.

They would be more valuable for bitcoin mining most likely.

~~~
nwh
2700 servers probably wouldn't be worth much for that. It would be noticed
fairly quickly, given that only CPU mining would be available, and how
monitored servers usually are.

~~~
cperciva
If you can hide the fact that you've rooted a box, you should be able to hide
the fact that you're doing bitcoin mining. Worst case, run the mining in the
kernel idle thread...

~~~
nwh
The apocalyptic heat and power use might be a giveaway.

------
nnq
This smells really funny. They could have buried this instead of going to
court (with a 1+ year delay!) and _committing PR seppuku by making this public
and giving their clients a reason of distrust._ Now, this is indeed the right
thing to do, the guy shouldn't go unpunished and they should disclose their
security breach, but if they are doing it for the "right" reasons, why is it 1
year later?

------
aptwebapps
"Gisse didn't return a voicemail and e-mail seeking comment for this report. A
Court docket shows he is scheduled to be arraigned next month and gives no
indication he has entered a plea in the case. He's being held at the Harris
County Jail on $20,000 bond, a spokeswoman at the district attorney's office
said."

Wait, wait, just because the guy's in jail is no reason not to return
voicemail and emails!

------
efnx
Would be way cooler if he rooted 2600 servers ;)

------
d23
> "He did not access customer content," Pelanne told Ars. "We caught it well
> before he had any chance to do any of that."

> Given the rapid discovery, the malware was on Hostgator systems for less
> than a month.

Then yes, he did. If the malware was on there for more than a few days, I find
it extremely unlikely that at least _some_ data wasn't compromised.

------
mikeurbanski
Sounds like he knows Linux...

~~~
Shadow_Death
It's funny you say that because I remember their add. I made a jab at the
company and - 2 rep.. If they knew the company they would have +2 rep me haha

------
Shadow_Death
I'm not surprised with that company actually, hostgator is a joke and the
employees in Austin, Texas are too. I think the one thing that bothers me how
did he get the SSH key? The fact that he had it tells me that someone higher
up dropped the ball somewhere.

