
DigitalOcean Root Vulnerability in the Wild - goodwink
http://badassrockstartech.com/digitalocean-root-vulnerability-in-the-wild
======
gregd
Aren't we jumping the gun a bit by saying, "DigitalOcean Root Vulnerability in
the Wild"? I'm not yet comfortable with that conclusion...

~~~
readme
I agree. To me it sounds more like _root vulnerability on OPs system_. He did
not change his password after using the automated password reset, knowing the
password was sent in plain text.

~~~
goodwink
No I didn't change the password. Instead I rebuilt the machine from the base
OS image. Would you expect that a new system with the disk wiped would retain
the old root password? I certainly wouldn't. Where were they keeping it?

~~~
bpicolo
Whenever I do a rebuild on a droplet it sends me a new root password in the
email.

~~~
goodwink
I have ssh keys set up so it doesn't do this. Instead what it seems to do is
reuse your old root password for the new image.

------
brokentone
TL;DR: Dude's server got hacked with a password-based SSH login. Doesn't
definitively find source of hack. Doesn't like host security procedures he
could have circumvented (changing password, disabling password-login). Posts
on HN: ZOMG, ROOT VULN IN WILD!

Edit: Disclosure, I met one of the co-founders at SXSW. Seem like cool dudes.

~~~
goodwink
I think you missed the part where they retained a root password over a machine
rebuild.

------
goodwink
To answer some common questions about this article:

Yes, I know my setup was very flawed. It was a test machine and in the midst
of having automated config scripts written for it which were being tested.
This is why the machine was vulnerable to the attack, but this does not
diminish the importance of the underlying compromise it exposes in
DigitialOcean's setup.

The point of the article is that someone is able to gain access to
DigitalOcean password reset emails or a database of root passwords which
shouldn't exist, but seems to given that they set your root password back to a
previously reset value after you rebuild your server from the base OS image.

------
kamme
Too bad the author makes it sound like digitalocean has severe problems while
his setup was clearly flawed. The really sad part is that this kind problem
will unfortunately only grow as vps systems become cheaper and cheaper. People
with less knowledge will set up their own stack and not think of the
consequences... I wonder what hosting providers will come up with to tackle
this problem.

~~~
goodwink
My setup was clearly flawed and I acknowledged that in the message. Their
system /does/ have serious problems though since someone is likely
intercepting their password reset emails or else accessing their root password
database which shouldn't even exist. I'd say those are real problems
regardless of the poor config of the machine (which I readily accept my mea
culpas for).

~~~
kamme
Well, to be honest, every setup has serious problems, it just depends how far
you want it to go. 100% Secure doesn't exist and while it may look like bad
service from your point of view, truth is I've seen other vps solutions do the
same thing. The issue here is that people with experience in system
administration change the default password and set up key based authentication
and try not to rely on password management from others. It's a shame your box
got hacked, but immediately jumping to the conclusion the whole of
digitalocean has a root exploit in the wild is a bit much imho...

~~~
goodwink
I don't think you reads the whole article.

------
thomseddon
As soon as I saw the text "Your root password will be emailed to you" on the
bottom of the droplet create page I opened a ticket, here's how it went:

"Your root password will be emailed to you"

\-- start --

Me

 _Just seen this at the bottom of the "Create a droplet" page. You're kidding
right?_

Them

 _The root password is sent via email because it is the easiest and fastest
way to get a user online and running a virtual server.

We strongly recommend updating the root password after you login for the first
time.

We also have SSH keys support so you can add your SSH key to the server during
creation in which case no email is sent and instead the SSH keys are added
under the root user for more secure access.

Thanks,_

Me

 _Just added an SSH key and you're quite right, I retract my blunt reaction.

I must say however that I still think emailing them is a fairly terrible idea
and I'm surprised your not worried about being found liable for a subsequent
server hack.

That aside, thank you for your swift response and for pointing out I can use
my SSH key. I look forward to using DO more_

\-- end --

N.B. I actually received their response twice from different agents suggesting
it was a canned response

Frankly I don't even think passwords should be an option, as on AWS (not that
they're perfect)

~~~
danielweber
For lots of users, they have to send _something_. And while they could build a
PGP-whatzits system for the small fraction of their people who use PGP, those
people are already going to be immediately changing their passwords anyway.

Heuristics like "keys are better than passwords" or "don't email passwords"
are good heuristics, but they shouldn't become absolute rules.

~~~
viraptor
They don't have to send the password. Allowing users to submit their password
over https and storing only the hashed version is another option.

~~~
danielweber
Then you fall back to needing to send some kind of login token to the user
when they forget how to log in to the HTTPS site.

Onerous security policies can make things worse as people work around them
behind your back. You can require certificates but there's nothing you can do
to make sure that people don't put their keys in their dropbox for "safety"
because they don't want to get yelled at for losing their keys when a machine
crashes.

People shop for VPSs partially based on convenience of management. If you make
your system secure but inconvenient, you can lose the people who would would
most benefit from a "good enough to stop the automated SSH bot" policy.

------
namidark
You left root login enabled and kept passwords on. And you're upset you got
compromised? You can also enable keys in the control panel and you won't have
to deal with passwords being emailed.

Those are the first few things you should be checking when setting up a new
server (disable password logins and only allow keys).

~~~
goodwink
I'm not upset I got compromised, it was very likely given my configuration, I
acknowledge that. I did have keys enabled so the password wasn't email
initially, it was only emailed as a result of a root password reset. The
article is about what the compromise means about DigitalOcean not about my one
box.

------
zalew
> it is likely that if you reset the root password via the web interface and
> don't change it afterwards that you are vulnerable

shocker.

~~~
goodwink
The shocker is that this is true even if you rebuild the box from the scratch
image.

------
brianbreslin
Off Topic: I met the founders at SXSW, and was wondering what you guys thought
of their product? $5/month for a simple ssd vps seems like a good deal (was
thinking of running a single wordpress site off of it).

------
instakill
I think the lesson we're all learning here is that if you don't do your own
Sysadmin/devops then you're almost guaranteed to be dealing with imperfect
systems.

~~~
ohboyacomment
If you do your own sysadmin/devops, you are _certainly_ guaranteed to be
dealing with imperfect systems.

(It's very likely worse to do your own unless you have the resources to do it
right. Hosting companies have scale. Not defending DigitalOcean here, just
challenging your assertion.)

------
andyhmltn
Is it not possible your email was hacked? I know from my experience with DO,
they email you the root password.

~~~
goodwink
The article mentions that this was verified to not be the problem.

------
Goranek
This makes sense, i got my account blocked for spamming via SMTP, and i didn't
send a single mail.

It took 5 support letters to get my account unblocked, and the solution was to
block SMTP ports...

I didn't have anything installed !!! Clear Ubuntu image

------
martinced
" _The successful root login followed only one unsuccessful attempt"_

Once a system is compromised by a root exploit what makes you think that _any_
information that this system is giving is true?

While it may be likely seen the circumstances it is by no way certain. For all
we know it may have been a bruteforce attempt which, once in, got disguised as
a known-root-password attack.

As long as people are going to think that a compromised system is actually
giving true information about what happened we'll be in big trouble.

Or OP tells us that SSH login and SSH login attemps are logged automatically
on another server which hasn't been compromised and then it's a different
story...

~~~
goodwink
The brute force attack would have been disallowed due to fail2ban being
enabled.

~~~
ohboyacomment
For anybody considering fail2ban here, install keys and disable passwords via
SSH ( _don't_ remove the password from root as most naïve administrators
suggest, since most hosting shops will give you tty1 access; just set
PermitRootLogin without-password and PasswordAuthentication no. You keep a
back way into the system in the unlikely event that you lose all of your keys
and retain physical access to the machine). For an additional layer, you can
also remove all terminals except tty1 from securetty.

Once you're in that fairly secure state, fail2ban is _completely_ pointless
and serves only to reduce noise in your log. Which you should stop caring
about. I'm serious. fail2ban is a complete waste of resources and will peg a
core tailing your SSH log because it's a resource-intensive piece of software
that serves no valuable purpose. If you're worried about the resources of
writing a log, redirect the log off the box and prevent your local logger from
writing to disk -- which you _should be doing for security reasons anyway_ ,
as discussed in another comment elsewhere in this thread (how do you trust a
box's logs once compromised?).

While you're at it, don't move SSH to a separate port. There are interactivity
QoS defaults in most networks that expect SSH to be on 22, and the clever bots
will find your "cleverly hidden" daemon anyway. And wait until I tell you that
in the event of your "cleverly hidden" port 2222 SSH daemon crashing, one of
your local users can put up a trojan SSH daemon in its place without your
permission (which is why SSH is on 22 in the first place).

Just deal with the log noise. We all do on edge boxes exposed to the Internet.

~~~
tomp
Is there a way to rate-limit ssh password-based logins to your system (i.e. 4
per minute or so)? I mean, across all users, all IPs.

~~~
jrabone
Simple 3 attempts per minute rate-limiter:

    
    
      echo "  (eth0 - SSH rate check)"
      iptables -N SSH_CHECK
      iptables -A SSH_CHECK -m recent --set --name SSH
      iptables -A SSH_CHECK -m recent --update --seconds 60 --hitcount 3 --name SSH -j DROP

~~~
ohboyacomment
Well, that's omitting the rules required to get an SSH packet to that chain. I
also hope you've already ACCEPTed ESTABLISHED,RELATED so we're only
considering these rules on new connections, otherwise you've entirely locked
out SSH after the third packet.

Assuming that's true, now guess what happens when you do this during a crisis:

    
    
        for i in backup-file-{0..999}; do
           scp $i that-machine:/var/db/
        done
    

Unless you're muxing in your local SSH config, you will cause the firewall to
intervene on you during the absolute worst time, and you will have no idea
why. Promise.

I really wish we could reverse the trend of fiddling with iptables when it
comes to SSH. Everybody does it and nobody thinks about the ramifications.

~~~
asb
You are totally right, and it seems non-trivial to work around this. One
approach would be to whitelist an IP upon successful logon. Perhaps by
modifying /etc/ssh/sshrc to invoke iptables (though this is not executed as
root...). An alternative would be to use pam_exec and have that invoke
iptables to whitelist the IP (assuming it can find out the IP). But I don't
think PAM is even used if SSH private key authentication is used...

