
Postmortem: Server compromised due to publicly accessible Redis - zurn
http://kevinchen.co/blog/postmortem-server-compromised/
======
antirez
After understanding how much this attack was used, I modified the blog post in
order to prevent script kiddies from using it without a few changes, but still
it caused a lot more harm than I expected. Before Redis 3.2 I'll add security
features to prevent this from happening, while adding them in a form that
should prevent also people from getting annoyed by the security features. It's
an hard mix to get right.

~~~
tedivm
You posted a tutorial describing how to use your software to exploit servers,
and you're surprised that people used it? I really hope this was a one time
lapse in judgement and not indicative of how the redis team thinks about
security.

~~~
andrewchambers
The only fault lies with those who put the service on the open internet.

------
barrkel
In addition to what others say - first firewall, then lock down services in
turn - consider running OSSEC or another intrusion detection system if you are
concerned about intrusion.

But really this is the OP's problem: he ran a system open to the internet
without any apparent analysis of attack vectors. Server was compromised due to
incompetence - not developer incompetence, but operations incompetence, a
different discipline.

~~~
rmdoss
Can't agree more with the OSSEC recommendation (
[http://ossec.net](http://ossec.net) ) . Been using it extensively and gives
an early warning system if something is going on (both in terms of files
changes, backdoors and log trails).

------
kudosneste
This is also a good reminder why running services as root is not recommended.
If he had used a separate non-root account for Redis, the damage would have
been limited to that single user account. That's not a very good situation
either, but at least the attacker would not have been able to modify init
scripts and replace system files.

~~~
repomies69
> If he had used a separate non-root account for Redis, the damage would have
> been limited to that single user account.

Probably not. The attacker would have gotten access to the machine. As there
is a lot of software installed locally some probably have holes, which allows
for full root access.

So I don't really believe it makes much difference for serious attacker if the
attacker gets root access or normal access.

~~~
eeZi
It's not _that_ easy to get a local privilege escalation on an up-to-date
system.

------
Terretta
Popping antirez quote on the lunacy of running this "publicly accessible":

> _" From time to time I get security reports about Redis. It’s good to get
> reports, but it’s odd ... in a software which is designed (as we explain in
> our security page here
> [http://redis.io/topics/security](http://redis.io/topics/security)) to be
> totally insecure if exposed to the outside world."_

Regardless of the readme, the Lessons Learned should include both "know the
security posture of the services you run" and "don't run _anything_ publicly
facing unless you are forced to, and only then if hardened well".

~~~
zurn
The first paragraph says he made a mistake when uninstalling Redis, he didn't
have it running on purpouse.

Everyone is fallible, things tend to go wrong. Insecure defaults in software
hurt even expert devs (which this guy is if you look at his blog).

If your software is "designed to be totally insecure if exposed to the outside
world" you should not ship it with defaults exposing it to the outside world.
IOW you should not design software to be totally insecure by default.

~~~
dijit
but it runs and binds to 127.0.0.1 by default, what better default is there
for this?

~~~
tedunangst
binding to 127.0.0.1 by default would be much better than the actual default
of all addresses.

------
SFjulie1
Kss, being compromised by a script kiddie is bad.

But I am puzzled: a running process like redis is supposed to run in a chroot,
and /tmp is supposed to always be mounted with no exec, the chroot and process
should have been runned with seteuid/guid redis...

I normally have backend technologies confined in a jails with pf authorizing
only inbound & outbound traffic on the legitimate interface+port, and you need
no ssh access in a jail.

Looks to me there is a cascade of sysadmins wrong practice.

Devops and modern "full stack devs" seems to have lost their basics.

~~~
jlgaddis
I wear a "sysadmin" hat (among many others) but not a "dev" hat. Every
sysadmin I know who's worth their paycheck does the things you've mentioned
above (and others), every time.

Perhaps this is what happens when you take a "dev" with no real sysadmin
experience and make him/her responsible for these machines? It's like giving
my neighbor's kid with a small dirt bike the key to my Harley and sending him
out on the highway -- bad things are bound to happen.

It reminds me of the common solution that many "amateur sysadmins" have when
they encounter SELinux issues: turning it off. :/

~~~
SFjulie1
Selinux is crap. To complex to configure, and poorly configured by default.

process poorly packaged crash and nothing in the logs.

You disable it, see there was a warning about a poor permission and shout at
selinux for making you lose days worth of works by poorly logging.

Btw, I am a dev. But I have to sometimes go in production see servers because
I need to help production sysadmins troubleshoot the weirdness of todays mis-
performing easy to use solutions (selinux, systemd, rsyslogd, packaging).

When you have to recompile centos or debian packages to have them support
vanilla features and you are a dev, it is boring.

------
techjuice
This should have been run as a non privileged user, on private IPs, a local
socket or localhost along with a long and strong password, so only
authenticated connections would be able to be able to use it.

Since none of these memory store solutions (memcached, redis, etc.) should be
able to be connected to on public addresses it would be a nice feature to only
allow them to listen on private internal IP space or sockets by default and
fail to run if anything else is used.

This would help reduce everyone's attack surface by default and help prevent
misconfiguration by the user or 3rd party integration software.

~~~
Terretta
You can use arbitrary address space in a non-routed subnet. But it could
refuse non RFC 1918 addresses unless overridden.

------
dgregd
The most important thing to do after a server setup is to enable firewall. By
default block all ports on public IP and then open ssh, http and https. There
are tools like ufw, just few commands and your server is secure. This is much
better approach than securing all services and system deamons run on a server.

~~~
pixl97
>This is much better approach than securing all services and system deamons
run on a server.

Um, no. You should absolutely be doing both. Defense in depth is the only way
to remain secure in failure of any other component. A firewall WILL NOT
PROTECT YOU FROM THAT.

For example, you just put on the firewall and leave a daemon running on
localhost as root. The firewall allows connections from localhost by default
(as most do). An attacker finds a hole in your webserver and is able to run
scripts in the httpd context. The script makes a connection to the daemon on
localhost, exploits it, and now has root privileges. With root the attacker
modifies the firewall rules to allow access. You are now fully exploited.

Had your daemon been secured they would have only been a takeover of httpd and
daemon accounts and not a full system compromise. Privilege separation is
extremely important.

------
KyleBrandt
Considering the skills demonstrated in this analysis, you might discover
getting started with configuration management (puppet/chef) is easier and
might take less time then you expect (even in a fast moving early startup).
Might have totalled less time then this hack took up.

------
jessaustin
_Install things through the package manager when possible. This makes it
easier to uninstall the software cleanly._

Of course one uses the system package manager when possible, and a specialized
package manager like _npm_ in most other cases, but not all is lost when one
needs a recent version of some _autotools_ monstrosity. In that case just
install it using GNU _stow_. Then uninstalling is just an invocation of "
_stow -D_ " (plus deleting a single directory).

But that only makes sense for a box one physically possesses. Why on earth
would one reuse a VPS? Nevertheless, I'm glad that TFA did, because the
investigation was very interesting.

------
viraptor
> "So we’ll just delete these files, kill the running process, and be done
> with it, right?"

No, no, no, never! You can poke around the server to figure out how you got
attacked, but afterwards the only way to recover is to rebuild from scratch.
Someone had root on the system. For all you know, you're currently running a
recompiled version of kernel/init, which will recreate all the original
scripts when you're not looking.

> Lessons Learned

I'd also add: learn about Apparmor / Selinux and use them in production. Even
if redis was running as root, you could still prevent most of the attack (no
file writing, no execution, etc.)

------
revelation
How can you make a software intentionally insecure and then also bind to _all
interfaces_ by default?

This isn't very difficult, just make it bind to localhost by default.

~~~
pixl97
>How can you make a software intentionally insecure and then also bind to all
interfaces by default?

In the last 20 years I've been working on systems, that is the default on way
too much. It's only been in very recent history that things come more secured
by default. In the past popularity was strongly correlated with ability to set
up. 'Works by default with no security' is really easy to setup and reduces
the number of complaints that "product is too hard to use, when with $x
product instead".

------
AznHisoka
Is there a Saas that helps detect these open instances for you? Sort of like
the opposite of pingdom. I want an email when something is accessible because
I might have changed a firewall setting that might have accidentally opened
the port.

~~~
sandstrom
There is a tool called Detectify[1], which is quite nifty! Not sure if it's a
good fit in your case, but possibly.

[1] [https://detectify.com/](https://detectify.com/)

------
jonesb6
As always I think the answer is in the middle. You can't have a piece of
software that defaults to insecure and expect everyone to have the common
sense and skill to secure it. You rarely can release a piece of software with
hard security defaults and expect it to have enough usability to gain critical
mass / adoption / be popular.

I remember when I first learned web development and I absolutely had a belief
that the defaults would protect me from royally fucking up. Postgres security
defaults for example. And although this is obviously not true in reality, we
can't let ourselves overlook it just because.

------
sandstrom
A neat command is `sudo lsof -i -P` which will show something like this:

    
    
        java       1114 elasticsearch  IPv6  TCP localhost:9200
        nginx      1169      www-data  IPv6  TCP *:80
        nginx      1169      www-data  IPv6  TCP *:443
        dhclient   1620          root  IPv4  UDP *:68 
        sshd       1697          root  IPv6  TCP *:22
    

I.e. services listening on various ports. Knowing which services are running
plus a firewall as an extra precaution goes a long way to avoid issues like
this.

------
dethos
As it was said in the "background section" this is not a problem with redis
but a configuration/ops problem instead.

Either way, you should always check the open ports and listening services in a
given machine. In this case for a quick check you could use:
[https://www.isredisallsafe.com/](https://www.isredisallsafe.com/)

~~~
pixl97
>this is not a problem with redis but a configuration/ops problem instead.

Default system wide failure is a redis problem. Any modern daemon should come
with configuration files that a) don't allow the application to run as root
unless uncommented by user action, and b) bind to localhost. Further
configuration should be necessary to break the system as thoroughly as this
user did.

------
zurn
Related previous discussion:
[https://news.ycombinator.com/item?id=10537852](https://news.ycombinator.com/item?id=10537852)

------
unixhero
I think you did a very good job at dissecting this attack. Kudos

------
golergka
I'm not a server developer, but words "play around" in context of production
server do seem like a red flag to me.

------
eeZi
Try Ansible instead of Puppet/Chef, it's much easier to use!

------
atmosx
This is strange[1]. Why on earth would you have a Redis DB facing the
internet?!

[1]
[https://www.shodan.io/search?query=6379](https://www.shodan.io/search?query=6379)

~~~
kevinchen
Because out of the box, Redis binds to 0.0.0.0 (and many people don't bother
changing the defaults)

~~~
atmosx
I know, but it's not an excuse. The _config_ file is pretty self-explainatory,
it takes less than 3 minutes to read through the comments.

~~~
jlgaddis
Pffft, reading the manual is for amateurs and n00bs.

We're professionals, we don't need to RTFM.

