
Identical Droplets in the DigitalOcean: Regenerate your Ubuntu SSH Host Keys now - jlund
http://missingm.co/2013/07/identical-droplets-in-the-digitalocean-regenerate-your-ubuntu-ssh-host-keys-now/
======
agwa
SSH host keys are problematic on cloud servers, not just because of this
problem, but also because if the cloud provider does the right thing and
generates the SSH host key on the first boot, the key is generated when the
system has very little entropy available. The primary sources of entropy on
Linux are key/mouse input, disk latency, and network interrupts. There's
obviously no keyboard/mouse on a server, and in an SSD environment like
DigitalOcean, disk latency is quite uniform and thus useless as a source of
entropy.

Linux distros mitigate the cold boot entropy problem by saving some state from
the RNG on shutdown (on Debian, it's saved in /var/lib/urandom/random-seed)
and using it to seed the RNG on the next boot. On physical servers this
obviously isn't available on the first boot, and on cloud servers, the
provider often bakes the same random-seed file into all their images, so
everyone gets the same seed on first boot (fortunately this doesn't harm
security any more than having no random-seed file at all, but it doesn't help
either). What cloud providers should really do is generate (from a good source
of randomness) a distinct random-seed file for every server that's created,
but I haven't seen any providers do this.

~~~
sweis
Regarding entropy sources for guest virtual machines, the host can expose a
random number generator device to the VM. This is an option in KVM:
[http://libvirt.org/formatdomain.html#elementsRng](http://libvirt.org/formatdomain.html#elementsRng)

Modern Intel x86 processors now have a hardware RNG built in, so even if the
host boots without any devices, you have a source of entropy.

(Incidentally, KVM defaults to /dev/random which might create a DoS
vulnerability if a guest exhausts entropy.)

~~~
karlkatzke
We've run into the /dev/random problem when doing heavy SSL traffic. We had to
pipe in sources of entropy. It was a janky solution, but wasn't as janky as it
might sound.

~~~
dspillett
If you need more entropy for SSL traffic than a headless server that sees very
little I/O load has available, I've found
[http://www.vanheusden.com/te/](http://www.vanheusden.com/te/) useful in the
past. The ~800 bits/sec is produced in our tests isn't massive, but it was
considerably than the machines we used it on could generate from the kernel's
usual sources on these machines. It claims to produce valid results in VMs
too, and given the way it works it might be perfectly valid to scale is some
way simply by running more than one copy.

If you want more than that in a cheap Heath Robinson manner (Americans: think
Rube Goldberg if you are unaware of Heath, their work came from very similar
inspirations) then many SoC solutions have a built-in RNG of sufficient
quality for general cryptographic use and some solutions expose them easily.
If you enable the relevant module a Raspberry Pi can provide up to ~550,000
bits/sec when asked to, for instance, which if one-off cost is a factor beats
paying a few hundred dollars or more for a USB device providing the same sort
of rate.

------
Nux
This is not the last of the problems we'll have with "the cloud", but I guess
it's part of what makes it so exciting. :-)

Many people, especially beginners, make the mistake of leaving the same SSH
keys in a certain template or in a snapshot of a virtual machine that they
later use as a template.

There are a few files that you really, really need to wipe out from a wannabe
image template:

\- /etc/ssh/* key* (for reasons explained in the parent article. stupid
autoformatting, remove the space after the first asterisk)

\- /var/lib/random-seed (the seed used to initialise the random number
generator. this is the location on CentOS)

\- /etc/udev/rules.d/70-persistent-net.rules (so that the VM's new NIC - with
a new MAC - can use the same "eth0" name)

People who want to do this more exhaustively can have a look at libguestfs and
it's program virt-sysprep which does all of the above and more!

[http://libguestfs.org/virt-sysprep.1.html](http://libguestfs.org/virt-
sysprep.1.html)

------
mey
I must say, I'm impressed with how this was handled both by the original
researcher and DigitalOcean.

~~~
specto
Except I informed them of this issue in January of this year

[http://imgur.com/GTi2UxJ](http://imgur.com/GTi2UxJ)

Apparently they ignored it :)

~~~
kintamanimatt
That worries me far more than the actual security issue. Security issues
happen to everybody, but so long as too many don't occur, it's the response
that shapes my ongoing confidence in that company or product.

What's happening on the left side? Is that you or the rep?

~~~
specto
I blocked out my name, the rep is the one with a picture.

edit: actually I didn't realize there was a skype window up when I took the
screenshot, thanks for warning me...

------
rwmj
They should be using cloud-init or virt-sysprep[1] on new instances. In
particular, it is _vital_ that you give your new instances a unique random
seed (which virt-sysprep can do). Also that you provide the virtio-rng to
guests that support it.

[1] [http://libguestfs.org/virt-sysprep.1.html](http://libguestfs.org/virt-
sysprep.1.html)

------
rlpb
To avoid this kind of security problem, use providers that use official Ubuntu
Cloud images only. If Canonical haven't certified the Ubuntu images you're
using, then your provider could have done anything to them. You'll need some
other way to determine their competence.

Cowboy images like this are exactly the reason trademarks exist. Commercial
providers who don't get certification are in fact violating Ubuntu's trademark
by telling you that you are getting Ubuntu, when in fact you are getting a
modified image which is possibly compromised (such as in this case).

~~~
regularfry
How do I validate that my provider is _actually_ providing an official Ubuntu
Cloud image?

~~~
rlpb
Technically? I'm not sure that's possible. They own the hypervisor, so you
have to trust them. This is why the Ubuntu trademark is so important.

~~~
regularfry
If I have to trust them because they own the hypervisor, the fact they claim
to deliver a "certified" image buys me nothing in terms of security.

~~~
rlpb
That's demonstrably not the case. In this situation, it was only the modified
image that accidentally introduced a vulnerability. The official image does
not have this vulnerability. Had you been using an Ubuntu certified cloud, you
wouldn't have been vulnerable.

~~~
regularfry
I've got no way to check I _am_ using an official image, other than knowing
that at some point in the past, my provider bunged Canonical some cash for the
"Ubuntu Certified(tm)" sticker. Without knowing a lot more about how
certification works, that's not a particularly strong proof.

------
makomk
This is now one of the first things I check when setting up a new VPS or other
VM instance, because it's really common.

~~~
voltagex_
I think I need a list of things to check. This sounds pretty scriptable
though:

* SSH Host Key

* SSH Authorized Keys

* SSH PermitRootLogin

* Disabling password auth in favour of keys

* Security updates from the distro

* SELinux (maybe?)

Anything else?

~~~
patio11
a) Move SSH off port 22. (Really limited security gain when implementing the
rest of the suggestions, but saves spam in your logs.)

b) Software firewall via e.g. IPtables is generally the first thing I turn on
after rebooting SSHd with the new settings.

c) (Optional) Consider using an architecture where you have N boxes and SSH
only listens on a local interface on N-1 boxes, with the Nth box running
_nothing_ but your VPN. (This is also a good architecture choice for admin
consoles, folks. www.example.com resolves to a public IP, admin.example.com
resolves to a private IP, so even if they're technically speaking on the same
box/boxes you won't lose the admin console if someone unwisely uses the same
password for a WordPress blog somewhere.)

~~~
dsl
My honeypots have been seeing scans on 2022, 2222, 3022, etc. for years now.
You should be setting proper ACLs for 22 and not moving to another port.

~~~
voltagex_
ACLs as in allow/deny? What if I'm using a dynamic IP on my client? In that
case, should I just be using a "trusted" SSH gateway?

~~~
dsl
You should be using a VPN into a trusted bounce box.

------
sehrope
Generating fresh keys aside, one thing I do with our AWS setup is whitelist
the IPs that can connect to our SSH bastion host. This completely eliminates
scripted port scans of the SSH server and makes the auth logs much more
manageable.

If our IP address changes (eg. ISP assigns a new one for the cable modem) then
we just update the whitelist (and remove the old address). It's _very_
infrequent. I could probably count the number of times I've done it on one
hand.

It might not be the most scalable setup but at our small size with everybody
working from home it works great.

The only slight hitch is updating it when traveling but even that isn't much
of a problem. It takes a minute or two from the AWS console and its good to
go.

I recently took a look at digital ocean ($5 servers gives me ideas...) but
didn't see a firewall option similar to the security group setup in AWS. If it
does exist then I highly recommend it.

~~~
werkshy
With Digital Ocean you'd need to install a software firewall on the servers
themselves, there is no API-configurable network-level firewall. I used 'ufw'
which was quite easy to get started with (on Ubuntu), and replicated my AWS
security group config pretty quickly. I added the ufw config to my host setup
scripts so it happens automatically.

~~~
sehrope
The problem with the software one is you need a way to modify it when you
can't access the instance. With AWS you do it from the admin console or APIs.
If it's on the machine itself you'd have to know the IP to open up in advance
or have someone at home base do it.

~~~
jon-wood
If you're likely to be connecting from different locations then you're
probably better off having a VPN in a known location and routing connections
to your servers via that VPN, rather than fiddling around with firewall rules
every time you're in a new hotel.

~~~
aeosynth
That just shifts the problem - the VPN is vulnerable to the original attack.

------
druiid
One good thing to note is that any VM image using cloud-init (a package for
debian/rhel systems) should automagically generate a new host_key set for any
new system image. Basically if you build a system image for EC2 or any system
that uses the EC2 data format (like Openstack) for host instantiation, then
you should install cloud-init. It would prevent something like this.

------
joshmn
Now that it's said, I did notice something strange once.

I had loaded up an Ubuntu Desktop droplet with the purpose of checking
something out through the browser on the node.

The startup page was
[https://www.americanexpress.com/](https://www.americanexpress.com/)

Since when is that default?

Didn't think much of it at the time, but now... whoa.

------
scottlinux
I suspect this kind of thing happens with other companies, but can only
speculate.

Somewhat related: chicagovps gave me a 'fresh' gentoo vps, and the default
provided root password was identical to the original one from several months
ago. I assume it is one gentoo image with the same password (for all
customers)?

------
schappim
Props to the way you handled this. That's how you do responsible vulnerability
disclosures!

------
throwawayh4xor
Just verified this is also the case with at least some AWS-hosted servers.
Coupled with the fact that many people simply ignore the MITM warning that SSH
throws, this is scary stuff.

~~~
novaleaf
unfortunately, devops (like me) need to ignore these as part of our work
system.

i need to automate deployment / reprovisioning of 30 digital ocean servers. as
reprovisionined servers frequently use the same ip address, i always run into
this. for me I had to disable the check :(

~~~
ef4
Part of automating deployment is automating key management. Ignoring host key
checks is the worst possible solution.

You're actually better off just baking a shared key into your image. So long
as it's a key you generated yourself (not like this Digital Ocean scenario,
where the key came from the cloud host), only someone who has already rooted
one server can successfully MITM your SSH connections.

Whereas if you ignore host key checks entirely, anyone who gains control of
one network hop between you and your servers can own you.

------
joeblau
Great find. I came from a heavy security background and moved to SV where it
seems like security is an after thought. I spent many long days and nights
STIGing RHEL boxes so I can appreciate this find. Also thanks for letting me
know about Digital Ocean, their VPS looks promising and I think I might start
using it.

------
davidhollander
> _After you have run those commands, simply restart the SSH daemon so it
> starts up with the new keys in place_

I believe if your version of OpenSSH is up to date, sshd will read the host
key each time a session is opened and does not need to be restarted.

------
stevekemp
We ran into similar problems on the hosting side; another surprise can be the
debian-sys-maint password configure by the Debian mysql-server package.

------
foxhop
So you are the reason I started getting these error messages, I noticed the
change on June 2, great work.

If you are still reviewing salt, I just wrote a post about salt-cloud and
DigitalOcean that you should check out -

Create your own fleet of servers with Digital Ocean and salt-cloud:

[http://russell.ballestrini.net/create-your-own-fleet-of-
serv...](http://russell.ballestrini.net/create-your-own-fleet-of-servers-with-
digital-ocean-and-salt-cloud/)

