
Fishing for Hackers: Analysis of a Linux Server Attack - gighi
http://draios.com/fishing-for-hackers/
======
jwcrux
OP may also benefit from the use of an SSH honeypot. I use kippo
([https://code.google.com/p/kippo/](https://code.google.com/p/kippo/)) with
great success. It tracks all commands run, as well as keeps copies of all
downloaded files.

In addition, it limits available commands to a certain predefined subset,
allowing the host to prevent damage caused (e.g. a DoS attack in this case) by
the system being compromised.

~~~
gighi
That's an interesting project.

Would it have recorded also statistics like the connection activity?

Seeing all the UDP traffic, and being able to trace its origin to the "@udp1
39.115.244.150 800 300" command, received not via shell but via a TCP
connection, was pretty cool.

~~~
infruset
Ironic that the IP may now suffer another DOS attack because of HN trying to
access it after reading the article.

~~~
gighi
I anonymized the IP addresses in a consistent way before publishing the blog
post, so in the very worst case the DoS attack will go towards a completely
new host :)

------
dthakur
Great article. I had not heard of sysdig previously.

Based on the timestamps of the entered commands, I guess one of the takeaways
for the attacker is to look into config management tools (eg ansible) :)

~~~
pmh
Since you hadn't heard of sysdig before, you might also be interested in this
article posted[1] a couple of weeks ago: [http://bencane.com/2014/04/18/using-
sysdig-to-troubleshoot-l...](http://bencane.com/2014/04/18/using-sysdig-to-
troubleshoot-like-a-boss/)

[1][https://news.ycombinator.com/item?id=7622121](https://news.ycombinator.com/item?id=7622121)

~~~
nounaut
I hadn't either and thank you very much for that link!

------
fragmede
So a DO/Rackspace/AWS VPS with a guessable root password can expect to be
cracked in ~4 hours?

That's terrible!

AFAIK, AWS defaults to ssh-key logins with password logins disabled. Can
someone comment about Rackspace/DO?

~~~
wazoox
This is not limited to AWS/rackspace... Here's from my home PC:

$ uptime

    
    
      22:09:07 up 30 days, 12:19,  2 users,  load average: 0,17, 0,09, 0,07
    

$ sudo fail2ban-client status ssh-iptables

Password:

Status for the jail: ssh-iptables

    
    
      |- filter
      |  |- File list:	/var/log/messages 
      |  |- Currently failed:	1
      |  `- Total failed:	1757
      `- action
         |- Currently banned:	0
         |  `- IP list:	
         `- Total banned:	242
    

1757 attempts from 242 IP address in the past 30 days...

~~~
akx
My home NAS is also exposed to the internet.

    
    
        up 298 days, 20:42,  1 user,  load average: 0.00, 0.01, 0.05
    

"zgrep ssh auth.log* | grep -i failed" has no traces of any intrusion attempts
whatsoever, just me not being able to type.

The distinction is, though, that the SSHd on that box is running on a non-
standard port (220)... so that certainly makes a difference.

------
salibhai
Genius idea. Love it. Shared it with my favourite web host. I hope more
security companies think like you do and do this type of reverse-phishing on
the bad guys ;)

~~~
delluminatus
Most security companies do this; the term for a monitored, weakly-secured
server like this is a "honeypot". It's a great way to find out about exploits
in the wild, which is really valuable knowledge for every hat.

Having said that, it's amazing what OP could do with just one monitoring tool.
Very impressive.

------
collingreene
Cool article! A friend and I once did this but then recorded the commands
attackers ran and replayed them on a big tv in our office. We called it hacker
fishtank.

~~~
breakall
That sounds cool! Did you post the presentation anywhere?

------
euske
I had a similar experience as the OP years ago. Luckily, the attacker forgot
to erase a bash history so I could recover almost all the command lines. It
seems that most of these operations were pretty standardized; they first
downloaded a bunch of exploits from another cracked site, then in my case they
tried to run a local exploit to gain a hole in kernel (it was a Linux 2.4 box
whose privilege escalation bug was fixed just weeks ago). I had a patch
applied to the kernel so it didn't succeed. Then they started an IRC bot as a
disguised name (like /usr/X11/X or something) and left. I felt embarrassed but
in hindsight it was a pretty good lesson.

------
Dorian-Marie
For those who don't know sysdig:
[http://www.sysdig.org/](http://www.sysdig.org/)

------
kilolima
Fascinating! Two questions:

Wouldn't it have been better if the attacker had removed only the last few
lines recording his commands from the log files instead of the entire files?
Wouldn't the lack of continuity in the log files be very noticeable?

Also, is this a script running this sequence of commands or an actual person?

And, is there a log somewhere on the system of 'make' activity?

~~~
gighi
To answer your questions:

1) Yes, it would have been better but I honestly think this attack was
completely botnet-driven and the attacker didn't really mean to cover his
footprints too much: in the timespan of 10 minutes, he sent over 800 MB of UDP
traffic. That would have been caught even by the most oblivious sysadmin
pretty quickly, so these guys are just playing a number game, trying to break
in as many hosts as they can knowing that the lifespan of the hacked hosts
will be very short, maximizing the short-term profit then.

2) The attacker directly ran these commands on the login shell (no script was
copied over scp or something else), so there was no script executed on the
host itself, but the whole thing lasted roughly 2 minutes and a lot of
commands were "typed", so I am almost sure this was just an automated script
ran from another probably compromised host.

3) I didn't check if the build left logs, but by showing every executed
process with "evt.type=execve" (which goes deeper than the spy_users chisel)
you can see all the processes executed by the build: 99% are just
uninteresting sed/gcc/autoconf.

------
eyuelt
I'm new to this area but am really interested. What do people usually use
other than sysdig to try to see if/how their machine has been compromised?

------
johnnymonster
Love this kinda stuff! Glad that there are people out there actively looking
for the latest and greatest threats to the internet.

------
user826
Awesome post! One question... Are you mounting your S3 bucket on the server?
If so, are you using s3fs? Thanks!

~~~
gighi
Yes, I mounted the bucket using [https://github.com/s3fs-fuse/s3fs-
fuse](https://github.com/s3fs-fuse/s3fs-fuse)

------
DodgyEggplant
Great article, thanks. Does sysdig ready for production servers? tested so it
does not introduce new issues?

~~~
degio
I'm one of of the sysdig creators. Sysdig is a pretty young project (we
released it around a month ago), so I can't promise it will be flawless in a
production environment. However, we've had many installations under several
different environments, and during the month after the release we had
extremely few crash reports, which we've worked to fix right away.

~~~
peterwwillis
By 'crash reports' do you mean the kernel/host?

~~~
degio
I mean overall, so that includes kernel crashes.

Kernelwise, the main thing to report is a couple of crashes on non-mainline
kernels like openVZ.

------
cosud
Wow a Romanian script kiddie at work. This kind of modus operandus is so 2004.
Nice tool showcased though.

~~~
itistoday2
_(Since HN isn 't letting me post a reply under chippy1337's comment, I'll
post it here.)_

I found chippy's comment interesting and helpful and don't know why it was
downvoted to hell while other "+1"-style comments (that didn't add any value)
are left as-is:
[https://twitter.com/taoeffect/status/464090445677481985](https://twitter.com/taoeffect/status/464090445677481985)

------
acdanger
'live capture not supported on OSX' \- not surprising, I guesss.

------
coreymgilmore
Great read, definitely gave me some ideas for securing my systems.

------
security_guy
Good job! sysdig quite a handy tool.

------
weishigoname
Interesting article, nice idea!

------
iplaman
good job!! nicely done!

------
pwelch
Great article! Thanks!

------
Gepser
¿But how did they enter? ¿Force attack?

~~~
gighi
Yes, I didn't put it in the article because it was getting too long otherwise,
but the attacker immediately tried brute-forcing the root account, and after a
handful of common passwords ("qwerty", "qwerty123", "pizza" among those) he
found "password".

I was able to find all the attempts by looking at the I/O activity of the sshd
process, and also the syslog activity recorded every attempt.

~~~
leaveyou
Doesn't your system refuse root login by ssh by default ? If I remember
correctly, on ubuntu server, sshd is configured by default to not allow root
login from remote addresses.

~~~
gighi
On some providers yes, in fact I explicitly enabled root SSH login for those.

Other providers (such as Digital Ocean) use the root account by default even
for Ubuntu, although the password is set to a really secure and random one.

