

Cache on Delivery (Mining random people's memcached) - jackowayed
http://www.slideshare.net/sensepost/cache-on-delivery

======
tptacek
I like the Sensepost people, but can sum up this talk very succinctly:

* You need to make sure you've firewalled (and/or on single-host configurations bound it exclusively to lo0) memcached so people can't talk to it from the Internet.

* If you survey the Internet, you'll find a lot of people who have not obeyed this rule.

* If you don't obey this rule attackers can get and possibly alter most of your data; an exposed memcached is probably game-over for your site.

The only things you should be able to talk to on anything in your data center
are ports 443 and 80, plus 25 on your single mail relay if you have one, plus
22 (heavily filtered) on a single relay SSH server. If you can talk to more
than that, you've probably done something very wrong.

~~~
tomjen3
Personally I am so paranoid with SSH that I don't run it on the default port.

That alone allows you to avoid most attackers trying to guess valid server
ips.

Now to put a honey pot on port 22.

~~~
KTamas
Oh yes, the first thing I do is change the SSH port, eleminates 99,99% of the
auth attacks.

~~~
Vivtek
Damn. Why didn't I ever think of that? I'm doing that tomorrow.

~~~
dugmartin
Remove root login over ssh and disable password logins and you are good to go.
If you want to get fancy Google for ssh port knocking.

~~~
mrduncan
_disable password logins_

I can't emphasize this one enough. Unless you need to login from a lot of
different machines, there really isn't any excuse not to do this. It also has
the bonus of making logins really easy since you don't have to type a
password.

~~~
InclinedPlane
If you host your systems on a VPS service like linode or slicehost then you
have the backup of a web based console in case you screw up royally and have a
HD crash on the one machine your SSH key was on (for example).

------
werrett
I've come across unprotected Memcached deployments in a couple of pen-tests.
On one engagement in particular someone poking Memcached would be able to
(temporarily) increase their account balance and even access CC numbers from
recent transactions. :(

As far as I see it, this is one of the unintentional side effects of "hosting
in the cloud". If you had co-located servers you'd whack up a firewall and
only allow your internal IPs to access non-HTTP ports. Alas everyone now just
spins up an S3 image and palms it off to Amazon.

~~~
bdr
With EC2, you have to opt-in any ports open to the public.

~~~
werrett
There you go, I stand corrected.

Are you able to make requests between instances on non-public ports? As
someone else pointed out Memcached infrastructure typically won't sit on your
local webserver.

~~~
fleitz
Yes, it's pretty easy to set this up with security groups, you can restrict
ports to only open to machines within another security group.

So lets say you've got memcached, mysql and a bunch of webservers.

On the memcached security group you open 11211 to the webservers group

On the db security group you open 3306 to webservers

On the webserver group you open 80 and 443 to everyone.

------
mikeklaas
How can people not run a restrictive firewall on their externally-accessible
boxes? EC2 even has this built in as a network-wide feature. Our production
servers run with the following setup:

* block everything from outside internal network

* open port 80 on web server

* open port 22 on all boxes, but only allow key-based authentication. Oh, and only allow connections from an ip whitelist.

The restrictions on port 22 are probably a little overkill, I admit.

~~~
maukdaddy
Restrictions on port 22 are NEVER overkill. In fact, you should run another
port entirely.

~~~
_delirium
If you're already using key authentication, that doesn't add much security. A
port is just a very short password shared by all users--- instead of logging
in with "I'd like to login, please initiate public-key authentication", you
now have to login with "I'd like to login, the magic number is 2222, and
please initiate public-key authentication", which isn't really any more
secure.

~~~
peripitea
Call me an engineer, but I don't understand why anyone would dismiss security
through obscurity just because it's theoretically useless. If you can reduce
the lifetime expected value of successful attacks on your system, it should be
considered a win regardless of how you do it. (Of course, relying solely on
security through obscurity is a sure-fire way to greatly increase said EV.)

------
sensepost
The blog entry on the topic is available at
<http://www.sensepost.com/blog/4873.html> Also, GoWalla & Bit.Ly were informed
pre-publication, and fixed their stuff.

------
lsb
Memcached by default only connects to localhost. Were these people using it
like that and/or does that matter?

~~~
yanowitz
In most deployed configurations, you use memcache in a distributed fashion (so
N machines can connect to a pool of caches), so you usually have them bind to
a public interface. One that you stick behind at least one firewall.

Exposing memcached (or riak or whatever) is operational malpractice.

~~~
_delirium
It's definitely dumb, but people might be lulled into thinking it isn't
necessary because the trend among other servers is to essentially include
their own firewalling setup, with a default of blacklist-everything that you
have to explicitly override by whitelisting IPs, or by having clients
authenticate. For example, you used to rely on firewalls to make sure nobody
was sending spam through your internal mail relays, but these days most MTAs
handle that filtering themselves, taking whitelists of "relay from here is ok"
IPs in their config files, even though in theory iptables/ipfw/etc. could
handle that job just fine, and more efficiently. More relevantly, most RDBMSs
have a similar setup--- they won't just let any client talk to them, even when
bound to a public IP.

For efficiency, though, memcached explicitly decided not to handle that kind
of filtering itself, e.g. by having the memcached config file take a list of
IPs of other memcached servers to talk to (defaulting to none), or including
some kind of auth mechanism, and instead relies on the system firewall for
that. That's a reasonable design decision, but increasingly unusual. Most
servers these days see the system firewall as a backup level of defense, not
the primary one, and aim to be secure even when unfirewalled.

~~~
macca321
I lulled

------
davidfischer
This was one of my favorite briefings at Blackhat.

