
Hacked: Investigating an Intrusion on My Server - frantzmiccoli
https://thedarkside.frantzmiccoli.com/tricks/2016/08/27/hacked-investigating-intrusion-on-server.html
======
viraptor
This article was absolutely painful to read. Let me try a different ending:

> Conclusion:

1\. Ensure the files that php is running are not writable by the same process.
Different app -> different user.

2\. Unless you're planning to send emails from the server, firewall output on
those ports. If you do plan to, firewall everything apart from that server
(you can setup alerts when DNS changes, it's not going to happen often for
email hosts)

3\. Disable most unnecessary functions and modules. Anything touching eval,
str_rot13, exec, and many others should be killed right away.

4\. Enable basedir.

5\. If you can't handle regular system updates, don't run your own server. If
you can't handle wordpress updates, host it with someone else.

6\. Wherever you host, make sure your app can be easily redeployed. You can't
rely on dates to see when anything changed.

7\. If you can't handle system upgrades, don't think that docker or any
software is going to solve any of your problems.

~~~
wtracy
I've never worked with PHP before, so basedir meant nothing to me. A minute
with Google turned this up:

[http://php.net/manual/en/ini.core.php#ini.open-
basedir](http://php.net/manual/en/ini.core.php#ini.open-basedir)

I have to say, for all the general brain-deadness we hear about from the PHP
community, the core developers have made a lot of progress on cleaning up
their mess. I don't know of anything equivalent to this functionality from
Python or Ruby.

~~~
malux85
I think basedir like functionality is a good idea, but this is not the
languages responsibility, this is an operational concern.

There's too many ways around it if you're trying to do this in a language ...
for example, limiting checks on the open() call would be relatively straight
forward, but what about people who use syscalls directly from the language? Or
what about shelling out to another process?

Chroot jails get kind of close of this in a very broad/blunt way, but back to
my original point: This is not the languages responsibility

~~~
viraptor
PHP doesn't have direct access to syscalls. (not without extensions anyway)
exec / system / other shelling out functions should be disabled unless you
actually know you need them.

------
efoto
In short: an old server running un-updated software was broken into not once
but at least four times since the start of the year.

The post describes some steps the author took to investigate and block the
attack. It's an entertaining reading, but I'd strongly advise against trying
this at home: it is not worth the risk. Reinstalling the system from scratch
instead would have been much more prudent.

~~~
voltagex_
Yep. "Treat servers like cattle, not pets".

I won't recommend Docker but something like Ansible can be used to make
reproducible servers.

~~~
redbeard0x0a
I think running a Docker container for Wordpress or Joomla installs. If setup
correctly with the right folders setup as volumes (i.e. wp-content for WP) and
the rest would be part of the docker image. If WP gets corrupted/infected,
then starting a brand new image would remove most of the corrupted files.

------
rmdoss
Fun read. If you ever get hacked, I recommend just destroying the server and
starting from scratch and a clean backup.

If you try to find all backdoors and left over rootkits, you will end up
forgetting one and being re-compromised.

~~~
z3t4
You can learn a lot by investigating how the attacker got into your system.

~~~
startling
Sure, but you shouldn't continue using the system.

------
0xmohit
One may find the following resources useful for measures on how to secure
one's infrastructure:

\- Security Guide: How to Protect Your Infrastructure Against the Basic
Attacker [0]

\- 7 Security Measures to Protect Your Servers [1]

[0] [http://blog.mailgun.com/security-guide-basic-
infrastructure-...](http://blog.mailgun.com/security-guide-basic-
infrastructure-security/)

[1]
[https://www.digitalocean.com/community/tutorials/7-security-...](https://www.digitalocean.com/community/tutorials/7-security-
measures-to-protect-your-servers)

------
AReallyGoodName
Question about this:

It's not possible for me to track all the 0 days for every piece of software
and library that my servers run. One of my long running servers could have
been backdoor-ed by a 0 day 6 months ago and i probably wouldn't know it. The
servers are kept updated but 0 days don't care about that by definition.

What's the best practice here? Should we pre-emptively have our servers
rebuild daily just in case a 0-day backdoored them?

~~~
viraptor
> What's the best practice here?

Defense in depth.

Remove the unneeded code so it never executes. If it's needed, block features
you don't need. For the rest go one level higher and block operations you
don't need. (php.ini controls, file permissions, LSM, etc.) For operations
that are allowed, ensure data/code separation to prevent persistence. When
that fails, have a very restrictive firewall so that data extraction and
attacking other systems is difficult. Have network restrictions again one
level higher. (switches / cloud network) If everything fails, have detailed
logs to know what failed and when. If even that fails, have backups. If that
fails, have company insurance prepared.

> Should we pre-emptively have our servers rebuild daily just in case a 0-day
> backdoored them?

Depends on your threat model and available money. Are you trying to stop
automated malware or state actors? In the first case a good configuration
should be enough. In the second you may want to start thinking about how
you're going to verify the firmware your harddrives are running. And there's a
whole spectrum in between.

I mean, to block majority of php blog/cms malware it's enough to make sure
your code is owned by user A, php server runs as user B, and B cannot write to
A's files. 0-day or not, lots of stupid automated malware stopped.

> It's not possible for me to track all the 0 days for every piece of software
> and library that my servers run.

That's why it's nice to run unmodified distro software. Because these guys can
track it for you: [http://www.ubuntu.com/usn/](http://www.ubuntu.com/usn/)
(for example, any reasonable distro has similar announcements)

------
schwede
You should consider running this malware detection script[0]. That script is
designed to catch malware php scripts. I've used it before with lots of
success. You will also want to check the access logs for all php scripts hit
in the last few months. Your hacker's spam script is likely being
activated/ran by a GET or a POST to that script. That's a pretty cheap way to
screen for other compromised files.

[0] - [https://www.rfxn.com/projects/linux-malware-
detect/](https://www.rfxn.com/projects/linux-malware-detect/)

------
cm2187
I did get one of these OVH spam warning emails too once. It was for a failover
mail server. It turned out that the software I was using (smartermail) wasn't
using TLS by default so the link between the failover and the primary server
was unencrypted. Spammers often target failover over primary servers. And OVH
was interpreting the failover passing on the spam back to the primary server
as me sending spam.

------
Neil44
Don't delete the outbound mail queue as they have a header in them which tells
you which php script sent them.

Also in the httpd logs look for POST 200's to scripts that shouldn't be
getting them, and look for long query strings esp with base64 in them. Check
those scripts and also scripts messed with by the same user-agent and ip's.

Never just delete anything, always check the contents and write a script that
can search and nuke it en masse. e.g. Find | xargs | sed etc.

When I had issues earlier this year I had postfix outbound throttled right
down, and a script to shut it down completely if the queue got big. So if I
got the monitoring email about smtp going down I'd know I'd missed something
;)

Ps, wp-cli, you need it in your life :)

------
mboelen
Thanks for sharing your intrusion story. Although it is always painful for the
person to experience, it gives good insights on what happens in the wild. As
the author of security tools rkhunter and Lynis, it is still a sign that basic
(and advanced) system hardening remains needed for a long time.

The other comments show that there are so many things you could do. But some
are more relevant than others. That is something we want to test for in Lynis
with upcoming updates. One focus area will be detecting
Drupal/Joomla/WordPress installations. If anyone wants to help out the open
source project and make the web a safer place, this is your chance to help:
[https://github.com/CISOfy/lynis](https://github.com/CISOfy/lynis)

Thanks again for sharing!

------
fake-name
Ahhh PHP, where `assert()` `eval()`s strings passed to it.

------
hwh
I wish it was easier to convert "standard" dynamic web pages into static HTML
easier. There's one or two Wordpress sites I'd really like to get rid of
updating over and over again (which is still less work than actually reading
mostly meaningless Changelogs and making a decision if there are security
issues that actually apply to me). Probably, my solution to this problem will
be "wget -r", plus a bit of sanitizing.

~~~
unkoman
There are plugins for this - Simply static, staticpress and more.

------
Hogg
Does }__ appear in your logs? All versions of all branches of Joomla prior to
I think 3.4.6 had a problem with serialization that allowed arbitrary PHP
execution.

------
chmike
Here are my advice:

1\. once compromized, start with a fresh copy. Use ansible to define automatic
installation scripts so that the server can be fully reinstalled in a "snap".
Use backups of the user data to restore.

2\. Automate upgrades. It's a piece of cake with ansible.

3\. Stay away from php. There are too many potential loopholes. I now start
using go to implement my web servers. Especially if the web site is static.
Check Hugo or tools like that. I implement dynamic servers with go and Iris
that uses fasthttp.

~~~
noinsight
> I now start using go to implement my web servers.

Then again, you can write buggy/insecure code in any language.

I don't think I've seen any articles about potential security pitfalls when
writing web services (or code in general) in Go. I'm actually interested if
there's anything surprising out there.

~~~
chmike
The difference is that we have a better control on what is being done when
using a statically typed language. Of course it is always possible to create a
vulnerability too.

------
waltwalther
There is always something to be learned. Thank you for sharing.

