
Kids, this is story of How I Met... my VPS hacked - lelf
http://www.corrspt.com/blog/2014/01/18/tale-vps-hacked/
======
zedpm
I'd think twice before declaring victory; I don't see any reason to believe
that the server is now "clean". If this had happened to me, I'd spin up a new
VPS, configure it appropriately, then install my app and migrate any needed
data. What I wouldn't do was continue running on an instance that had been
exploited and assume I'd successfully cleaned it up.

~~~
laumars
Agreed, particularly when he's already admitted this was the second time.

However I still praise him for not only taking time to investigate how the
breach happened, but also to blog about it so we can all benefit from the
experience.

~~~
zedpm
Oh yeah, I agree. I guess I failed to address that in my original comment, but
the analysis is actually cool and it's a valuable submission.

------
drdaeman
And the real issue was (is?) the fact that web application process had write
access to a location it was able to execute code from.

It is well-known for decades (at least since PHP got popular, which was in
'90s!) that such locations should only be writeable by the user(s) who
maintain the code and not anyone else. Sometimes, such setup can't be done on
dirt-cheap FTP-only shared hosting services ( _shrugs_ ), but certainly not on
VPSes.

Ignorance is bliss.

~~~
jlgaddis
Yep, this can be a lesson to developers who read this. If your web application
must write to files, keep them outside of the DocumentRoot!

edit: I know a lot of HN'ers are fans of Ubuntu but, FWIW, SELinux (default on
RHEL and derivatives) would have prevented this.

~~~
drdaeman
Outside of any executable location, not only DocumentRoot. Latter is just an
easier and more popular target.

I mean, all writeable directories (if any) must not only be not runnable by
some server (like Apache with mod_php), but also be on devices mounted with
noexec, like this is the case with /tmp.

~~~
jlgaddis
I'm not sure that'll help like you think it will.

I can't verify it at the moment, but create a /tmp/phpinfo.php file with
"<?php phpinfo(); ?>" in it and then run "php /tmp/phpinfo.php". Regardless of
whether /tmp is mounted noexec I think you'll find that it executes (because
the php binary is under /usr which almost certainly isn't mounted noexec).

~~~
drdaeman
Whoops, seems that you're right, I was wrong about interpreters taking noexec
into account - either I misremembered something or things changed, but the
fact is they don't. Sorry.

Anyway, it's also better to have noexec than not. :)

At least, it'll help if there's no complete shell access but only ability to
run some binary by name, without args, or modify environment variables when
running some executable.

------
pwnna
Ugh. I totally thought that someone met his/her spouse after some his/her VPS
was hacked.

Now I'm slightly disappointed. Am I the only one? :\

~~~
shill
It's a very odd title.

~~~
systematical
I'd go one step further, its a crappy title.

------
joshbaptiste
Long story short, he was running an instance of JBoss that had a vulnerability
that allows an attacker to execute commands as the Jboss user and ended up
Bitcoin mining on his VPS. Isn't Jboss one of these bloated “enterprisey” JVM
container frameworks? I would think something like Play/vert.x would suit the
likes of a small VPS better.

~~~
drdaeman
Choice of framework is not that important. The real problem was (and, judging
from the article it's still not fixed) that his server has write access to
directories it can execute code from.

I mean, this is a classic example of badly configured filesystem permissions.

~~~
deckiedan
This is one of the things that scares me about deploying PHP apps. I'm being
asked to take over hosting a couple of Joomla & Wordpress sites at work, and
find it terrifying that they both ask for permission to install php scripts. I
much prefer having a clear separation, but it seems that that isn't really an
option.

~~~
dclara
Yes, PHP has weaker security but fast implementation because it allows direct
database access from the front pages. There are a lot of discuss about it. I
always disable PHP on the HTTPD to avoid potential issues.

------
xhrpost
I feel an often overlooked prevention for these semi-random attacks is a
change in the SSH default port. I've posted this idea elsewhere and people
seem convinced that this is pointless because an attacker can just port-scan
your machine. While true, this generally only happens when you are being
specifically targeted by an attacker. More random attacks like the one
mentioned here are likely just people scanning all the IPv4 space looking for
open 22 ports and then testing known exploits. Since I don't run a super-
popular site, I'm more likely to be the victim of the second kind of attack. I
used to have bad-logins hitting my box on a regular basis, after switching
ports for SSH, the log-in attempts went to zero.

~~~
sliverstorm
I stick with iptables IP whitelists. We're talking /31 or /32 whitelists. I
really don't need SSH access 24/7, so it's fine having just a couple chairs
permitted. I don't need to SSH from the airport.

That's naturally not the only layer of security, but I figure it's a nicer
option than non-default port.

~~~
Nick_C
I have four VPSs, mostly on different hosting companies. 99% of ssh attacks
come from China, the rest are eastern Europe (mainly Romania or Ukraine) or
Brazil.

The Chinese attacks use a group of about 15 IP addresses, then, every so
often, they all change the addresses to new ones at once. This has just
happened, last week, in fact. So now I have dozens of attackers all coming
from a group of about 15 IP addresses, which are different to the 15 or so IP
addresses they used a couple of weeks ago. (No kidding, the regularity that
this happens, it would not surprise me if their military is training a new
class of crackers and has been assigned a different set of addresses to use
this term.)

When I get a new IP address in the log, I do a whois and rewrite the
"inetnum:|NetRange:" field to a class A|B|C address and then DROP it in
iptables. Fuck 'em. The whole darn network class gets dropped. Not that I'm
likely to be logging in from China any time soon anyway.

I now have a list of network classes with about 35 address ranges that get
dropped, if anyone is interested in the list.

------
jliptzin
Based on that pool address the attacker seems to be mining Batcoin, not
Bitcoin, which is appears to be an scrypt-based altcoin, far more profitable
to CPU mine than Bitcoin. (In case anyone was wondering why an attacker was
bothering to mine Bitcoin with a CPU).

~~~
phaed
Came to say this. Indeed this has nothing to do with Bitcoin.

------
chippy
Always useful to see attack vectors. Nice well written blog post.

~~~
tinco
The attack vector is some random part of the application server stack (JBoss)
demarshalling and running commands on strings received directly from web
clients.

A rather horrible practice to begin with. There's always so much that can go
wrong when parsing tainted strings to native data structures, let alone to
full objects using the builtin marshalling functions, they're not meant to be
used on objects from untrusted sources at all!

~~~
slig
Not long time ago Rails had an attack vector very similar to this, IIRC.

~~~
meowface
Yep, same thing but with YAML deserialization. Deserialization vulnerabilities
are common for Java, Python, Ruby, and PHP web apps, because deserializing an
untrusted input is nearly akin to running eval() on an untrusted input.

~~~
ambrop7
1\. Define exactly what the deserialization output should be. 2\. Implement it
that exactly that way. Now it's simple. Does the definition in part (1)
include execution of arbitrary commands?

~~~
Robin_Message
A deserializer might be able to instantiate arbitrary classes, so any class
with a constructor that could execute an arbitrary command makes the
deserializer vulnerable.

Of course, the correct answer is not to use the deserializer that can
instantiate arbitrary classes when you have a well-defined list of classes
that can be instantiated.

------
belluchan
I pasted a gist of the backdoor script the hacker downloaded onto that server:
[https://gist.github.com/anonymous/8527149](https://gist.github.com/anonymous/8527149)

Original URL is [http://pdd-nos.info/.tmp/back.conn.txt](http://pdd-
nos.info/.tmp/back.conn.txt)

------
nogridbag
Not loading for me. Here's Google's cached version:
[http://webcache.googleusercontent.com/search?q=cache:ZbXfxz_...](http://webcache.googleusercontent.com/search?q=cache:ZbXfxz_SkcYJ:www.corrspt.com/blog/2014/01/18/tale-
vps-hacked/+&cd=1&hl=en&ct=clnk&gl=us)

~~~
joshbaptiste
And kids, here's another story of how my VPS got DDoS'd by Hackernews/Reddit.

------
Zirro
I'm interested in setting up a Linux-based server at home, but one of the
things that have kept me from doing it is security concerns. Are there any
server distributions which are very strict in regards to which packages are
included/enabled, and configured with as little remote access as possible by
default?

~~~
orthecreedence
Yeah, Slackware =]. People tend to laugh when you use it ("LOL WHO USES
SLACKWARE!?!??!111") but its packages are hand-picked to be stable and rock
solid. On top of this, it's very easy to get a bare-bones install that doesn't
enable a bunch of useless shit (and open ports) at startup. Beware, all
package management is by hand for the most part and doesn't support
dependencies. So if you install a package, you have to know what other
packages it depends on. Also, be ready to compile some of your favorite
services by hand since a lot of them aren't in the distributed packages (think
things like NginX, HAProxy, etc). Things like Apache, MariaDB, Python, Ruby,
PHP, are included though.

It's one the oldest linux distros and doesn't really hold your hand that much.
You'll be editing a lot of config files by hand. However once you get used to
it you'll definitely feel a much stronger connection to your box than with a
lot of distros that try to make things easier. /r/slackware is a good
resource, there are a good amount of lurkers that jump at any chance to help
someone out, and linuxquestions.org is a good forum as well.

EDIT: forgot to mention: don't rule out FreeBSD/OpenBSD as well. They are
known to be pretty solid for hosting as well.

~~~
agwa
Slackware is OK if you want to play around with Linux on a non-public-facing
system, but it's bad advice if you want to run a secure public-facing server.
It's too difficult to do automatic package updates on Slackware. Since so
little software is packaged, you end up having to install most software
manually, which means _you_ become responsible for monitoring that software
for security advisories and upgrading it in a timely manner when there's an
advisory.

Here's my advice for a secure public-facing server: use Debian Stable, set up
automatic upgrades every night, install _as much as possible_ from the
official Debian repository, and be sure to upgrade to the next Debian release
before your current release loses security support. This way, the Debian
Security Team is responsible for monitoring security advisories and rebuilding
packages instead of you having to do it yourself.

Source: I just finished transitioning to Debian after 10 years of Slackware
use, in large part because I found it too difficult to keep my Slackware
installations secure.

------
this_user
The problem with old JBoss versions is that their configurations were insecure
by default with several potential attack vectors available. If you didn't know
about this and just deployed your JBoss out of the box, chances were good
you'd get "hacked". Newer versions (>= 7) fortunately solved that issue and
are not that easy to take over.

------
marquis
Anyone running php, move the location from /cgi-bin/php. We've seen a few
takeovers with unpatched systems to run miners. Not everyone can take their
server down and changing the location according to apache/nginx at least stops
the bots from finding you immediately.

~~~
antihero
People are still running PHP as a cgi? I though the way to do it now was using
a PHP-FPM pool with minimal privileges.

~~~
marquis
Oh yes. There are some rather ancient things still out there that give me the
shivers, but you come across them time to time.

------
asdffdsajkl
> Things could have been worse If the attacker found a way to upgrade the
> privileges of the user running jboss (it’s a sudoer, but the password is
> really hard)

NEVER GIVE THE USER YOUR APPLICATION SERVER RUNS UNDER SUDO PERMISSIONS!

------
agumonkey
Was the attacker gathering results through remote logrotate communications ?

Reminds me of wordpress attacks, you quickly wish for FS diffs in order to
identify any change in your code/data ...

~~~
spydum
Tripwire can be used, you just need to add your webroot and update it. These
tools aren't new, just not often utilized with the new kids on the block

~~~
agumonkey
You're right, I found about it afterwards. I wonder why this hosting company
didn't provide it (or an equivalent) out of the box.

------
antsam
I hope he flattened that VM and reimaged...

