
The default OpenSSH key encryption is worse than plaintext - rargulati
https://latacora.singles/2018/08/03/the-default-openssh.html
======
tptacek
Spoiler: the default SSH RSA key format uses straight MD5 to derive the AES
key used to encrypt your RSA private key, which means it's lightning fast to
crack (it's "salted", if you want to use that term, with a random IV).

The argument LVH makes here ("worse than plaintext") is that because you have
to type that password regularly, it's apt to be one of those important
passwords you keep in your brain's resident set and derive variants of for
different applications. And SSH is basically doing something close to storing
it in plaintext. His argument is that the password is probably _more
important_ than what it protects. Maybe that's not the case for you.

I just think it's batshit that OpenSSH's default is so bad. At the very least:
you might as well just not use passwords if you're going to accept that
default. If you use curve keys, you get a better (bcrypt) format.

 _While I have you here..._

Before you contemplate any elaborate new plan to improve the protection of
your SSH keys, consider that long-lived SSH credentials are an anti-pattern.
If you set up an SSH CA, you can issue time-limited short-term credentials
that won't sit on your filesystems and backups for all time waiting to leak
access to your servers.

~~~
lima
And/or use a smartcard to store your SSH keys. With YubiKey 4 being as cheap
as they are, there's little excuses :-)

~~~
tptacek
This is what I meant by "elaborate new plans". I like Y4's as much as the next
nerd, but if you're going to put that kind of energy in, spend the energy on
setting up an SSH CA.

~~~
lima
I recently rolled out YubiKey 4s to my whole organization and it was a
painless experience. There's not a single file-based key left. Provision the
keys, replace the old file-based keys via our configuration management
tooling, done.

Wouldn't a SSH CA just introduce a whole different kind of complexity?

\- How to protect the SSH CA and its key and make it highly available? I don't
want to be locked out of my bastion host after something the CA depended on
broke and my certificate has just expired.

\- How to authenticate users against the CA? Most solutions I've seen use a
longer-lived client-side secret, which is just as susceptible to theft than a
regular SSH key, or some sort of OAuth or SAML SSO. A malicious Chrome
extension can now compromise the SSO process, you still need a U2F token (like
a YubiKey 4) to properly secure the SSO account, etc.

\- How to make it work with our SCM and random things like a storage appliance
and various JunOS devices, which support regular SSH keys, but don't know
about SSH certificates?

I would assume that Latacora is using a SSH CA, and I'm legitimately curious
how you approached these challenges.

~~~
lvh
You raise a bunch of valid points. Just to answer the one about authing
against the CA (I'm in the LV airport for BlackHat and my laptop battery is
about to die): yes, still get U2F tokens. You're right that malicious Chrome
extensions will mess you up, but that's true for everything else you run too:
you need to enforce Chrome extensions via MDM regardless of what your SSH key
story looks like. I consider having to SSO in a good thing: it means
onboarding/offboarding/audit logging is easier.

The context for SSH CA/Teleport is SSHing into a box. When you do actually
need an SSH key, Yubikeys are the best answer. (I like using gpg-agent's ssh-
agent emulation mode because I find it works better on Macs, but that's
irrelevant to the security analysis.)

~~~
lima
I agree that for a large organization - which has the necessary pieces in
place - it makes a lot of sense to use SSO for SSH access. The SSO is mission-
critical anyway, there might not even be direct SSH access except via an
authenticated proxy, there's centralized audit logging and intrusion
detection, ...

However, I would argue that unless this is the case, operating a SSH CA is
riskier (both from a security and availability point of view).

------
jakobdabo
A rogue npm package is my nightmare scenario, those packages are like the only
(or I wish..) untrusted code my OS is running.

That's why I came up with a small script/service waiting for an "inotify"
event on a honey-pot SSH key (and some other files like ~/passwords.doc) which
will immediately shutdown the computer on any kind of access of those files.

    
    
        #!/bin/sh
    
        inotifywait -e open --fromfile ~/.config/killer/killer.cfg --outfile ~/.config/killer/killer.log
    
        if [ $? -eq 0 ]; then
            poweroff
        fi

~~~
jwilk
I fear this might be too slow to stop an automated attack. Passing -f to
poweroff, so that shutdown(8) is not called, might help a bit.

~~~
brazzledazzle
You could also just down your network interfaces.

~~~
thisistomsacct
Don't forget about ransomware. Imagine the rogue package encrypted your files
and demanded payment to get the decryption key. With public key encryption or
a symmetric key that is securely deleted, there is no need for network access
for the attack to succeed. A fast shutdown could protect most of your files.

[https://en.m.wikipedia.org/wiki/CryptoLocker](https://en.m.wikipedia.org/wiki/CryptoLocker)

Although I think the best way to deal with ransomware is a strong backup
policy, not a tripwire shutdown trick.

~~~
jwilk
Non-mobile link:

[https://en.wikipedia.org/wiki/CryptoLocker](https://en.wikipedia.org/wiki/CryptoLocker)

------
quotemstr
Okay. Sure. We should move behind the kindergarten stage of KDF selection.
Still, look at the threat model. We're talking about the encryption of the
private key at rest. In ~/.ssh, 0700. If an attacker can even _read_ your
private key file, you've probably already lost, encryption or not. That
attacker can probably change your .profile to include a keylogger. You lose.

I mean, sure. Change the KDF default to something modern. But the threat we're
discussing is marginal, and it's not as if the security of the SSH _network
protocol_ , which is paramount, is under threat.

If you care about this class of attack, you probably care about it enough to
use an SSH CA anyway.

~~~
lvh
The blog post explicitly addresses this, but you don’t seem to interact with
its point. We have evidence that smash and grab attacks exist, and since they
affect more people, you’re more likely to get screwed by something like the
recent eslint-scope thing than a targeted attack where the attacker does shit
to your .profile.

That said; yes: you should have long-held auth on a hardware token and then
use an SSH CA for temporary auth.

------
codesections
This argument seems to hinge on the following statement that I don't
understand:

> an SSH key password is unlikely to be managed by a password manager: instead
> it’s something you remember.

Why is that the case? I've always been just as likely to use a password
manager for my SSH key password—after all, I'm usually prompted for my SSH key
password from my terminal, which accepts pasted passwords just fine.

Am I missing something?

~~~
CBLT
Here's my anecdata:

I didn't know much about ssh keys when creating, so I followed directions
online. The directions I followed created my key with no password. Soon my Mac
encrypted the ssh key by itself, without my intervention (!), to use my Mac
logon password. The encryption format was the old ssh format the article
complains about, even though my openssh supported the new one. The password I
use to sign into my computer is not grabbed from a password manager.

~~~
angry_octet
What? Are you saying macos changed the key to be encrypted SSH format? That
doesn't sound right. In fact, implausible.

------
simias
>You might ask yourself how OpenSSH ended up with this. The sad answer is the
OpenSSL command line tool had it as a default, and now we’re stuck with it.

Given that (as far as I know) the decryption is handled only on the client
side, why does an old default matter? Does it break any kind of compatibility
to change that? Why only use the better algorithm for EC keys?

And even if it did break compatibility it's not like it usually stops the
OpenSSH folks, old ciphers and hash functions are deprecated regularly (I have
to whitelist a bunch of ciphers at work to connect to legacy systems running
an old dropbear).

~~~
tialaramex
Some people move private keys. You shouldn't, but people do. And so they
expect that an encrypted private key file from their brand new Mac will work
when pasted into a five year old CentOS install.

~~~
simias
Seems like simply issuing a warning on key creation "this key uses a newer,
safer encryption scheme but as a result might not be directly portable to
older versions of openssh, if that's an issue use <someflag> to use the older,
weaker scheme instead" would probably solve the problem in 99% of the cases.
After a few years you might remove the warning and move on.

For this type of applications security should trump backward compatibility,
especially when in this case the breakage is fairly limited and easy to work
around.

------
tcmb
I'm assuming that practically everybody who is a developer on ssh-keygen knows
about this. This is not a bug, right?

It's basically bad usability, where the default is unsafe, and in order to
make it safe, you have to know what you're doing. Or is it like the author
writes in 'TFA' that "we're stuck with it", because it used to be a default in
OpenSSL?

Some people will say, "if you're using ssh-keygen we can expect that you know
what you're doing". But this is a wrong assumption that gets people into
trouble. I'd even say a minority of people who invoke ssh-keygen really know
what's going on in detail and what providing a flag like "-t ed25519" actually
does. I certainly don't, and I've had to generate quite a few ssh keys so far.

~~~
lvh
The new format has been around for a pretty long while (2013). I think it
makes sense to upgrade to -o. While it's obvious to developers (I hope!) it's
not obvious to most end users I think :)

~~~
claytonjy
not only does it seem like this is a better default, this option is awfully
well-hidden on my system; doesn't show up in the short help or the example
usage in the man page; gotta scroll down to find it in the exhaustive list of
flags

~~~
Avamander
The utility also does not warn about insecure defaults.

------
nebulous1
This is a decent article with a click-batey title.

The argument that it's _worse_ than paintext hinges on you both using a weak
password and using the same password for something else. If you don't do
either of those things then it's much better than plaintext, although
technically if you use the same strong password for something else then you
may have weakened the password by using it for old-format ssh ... but it still
has to be brute forced.

~~~
lvh
If you select from a 10k word list, you get 13.3 bits of entropy per word. 4
of those get you 53 bits of entropy. One p3.16xlarge does 450GH/s. So, as a
first order approximation: 2 __53 / (450E9) = 2E4 or about 6 hours. Sure: you
don’t get to keep that performance, but we’re talking a day. What kind of
passwords do you have that make it “much better”? Or are you saying that
already counts?

~~~
ascar
53 bits of entropy is basically the same entropy as a 9 character alphanumeric
password. That's not a strong password to begin with and makes this
calculation senseless. Redo that calculation with 71 bits (12 char
alphanumeric, still not really long) and it takes close to 200 years, thus
making it impractible.

The problem with these word lists is, that they yield rather low entropy
compared to random strings, when the attacker knows that a word list is used.
That's because one word has just a little bit more entropy than 2 alphanumeric
characters.

~~~
lvh
Do you have evidence that passwords for SSH keys are as good as you claim?

~~~
petee
...conversely, do you have evidence most people pick their passwords from a
10k word list?

And the Oxford dictionary has ~171k words, so not sure where a 10k list even
comes into play

~~~
angry_octet
I think haveibeenpwned is good evidence people choose weak passwords and reuse
them.

DICEWARE uses ~8k words, chosen so that people can remember and spell them.

~~~
petee
haveibeenpwned provides no such statistic, other than reuse (75% of original
set) -- a breached password is unrelated to its strength.

The simple fact that the HIBP password list contains 512 MILLION unique
passwords...

~~~
angry_octet
> a breached password is unrelated to its strength

But most of them were weak. And most of them have been cracked by _hobbyists_.

[https://blog.cynosureprime.com/2017/08/320-million-hashes-
ex...](https://blog.cynosureprime.com/2017/08/320-million-hashes-
exposed.html?m=1)

512 MILLION is not a large number of passwords to check. What do you think the
hash rate of modern GPUs/ASICs is?

[https://www.troyhunt.com/86-of-passwords-are-terrible-and-
ot...](https://www.troyhunt.com/86-of-passwords-are-terrible-and-other-
statistics/)

~~~
petee
You're forgetting the salt added. Compared to the original post I commented
on, 512 mil is way larger than 10k. So it's 512 mil _known_ passwords, times
however large the salt is...it's not trivial -- and their 10k list was
expected at 6hours...

Extrapolating, 10k:6hrs == 512m:35 years

~~~
angry_octet
A GTX1080 gets 25GH/s on MD5, that is 1.5x10^16 per week. The salt is known --
makes rainbow attacks impractical, but doesn't reduce the hashing rate.

Martin Kleppman explained the problem back in 2013:
[https://martin.kleppmann.com/2013/05/24/improving-
security-o...](https://martin.kleppmann.com/2013/05/24/improving-security-of-
ssh-private-keys.html)

~~~
petee
I conveniently forgot the salt is public ::facepalm::

I also was using the OP numbers, verses trying to do any math or research
myself first ::second facepalm::

~~~
angry_octet
It alright, that's why we have HN - not to be always right first and fastest,
but to go deeper.

------
jwr
I'd suggest following the excellent guide at
[https://github.com/drduh/YubiKey-Guide](https://github.com/drduh/YubiKey-
Guide) and using one of your GPG subkeys (kept on a hardware device like a
YubiKey) for SSH authentication.

------
lvh
I wrote this; happy to answer questions.

~~~
pvg
Beside compatibility, is there any reason _not_ to use Ed25519 keys for
everything?

~~~
lvh
Nope! Ed25519 is great and honestly I think the compatibility concerns are
overblown. It was introduced in 2014. If you are running four year old SSH you
have other problems, like probably RHEL or some nonsense.

That said: Ed25519 is a local optimum: the bigger win is to not have long-held
credentials at all, and instead use an SSH CA or something like Gravitational
Teleport. Doesn't work for GitHub though.

~~~
glenjamin
In a world where you don’t have long held credentials - how do you get the
short term credentials?

~~~
xyzzy123
We do this:

Master user records in an identity/SSO system. We use ADFS.

Use a protocol which provides a portable “signed”, temporary credential. We
use SAML but OIDC also works.

Have a command-line client which auths to the SSO and then relays an identity
proof to a trusted component (lambda, cloud function, dedicated instance,
whatever your policy says is ok). The identity proof is just a saml assertion
or oidc token.

Have trusted component validate the identity proof from SSO and generate time
limited credentials (we issue ssh, kube and iam). Our client then auto-xfers
those creds up to jumpbox but you could drop them on workstation instead if
your policy allows.

~~~
wvh
I currently work in academic environments and I don't have much faith in the
SSO systems. I thought about this, but often the SSO credentials are also the
email password – the one password that gets entered and stored just about
everywhere for the average user. I'm not willing to bet the security on a
user's main login/email password, which seems to be a common setup. With that
password or even just access to a user's main mailbox, a smart attacker might
get through multiple layers of security by simply convincing services or staff
to reset credentials.

Though it might work with a second, more limited (not-so-)SSO system for
specific users and environments.

~~~
lvh
You're right that shitty systems train users to be phishable -- by typing
their password all over the place -- all the time, not even counting the poor
default password hygiene most users have. This is one of the reasons
U2F/WebAuthn is so incredibly valuable.

------
AceJohnny2
How about using macOS's Keychain to store keys?

On recent versions, need this in your ~/.ssh/config:

    
    
        # for Keychain (acting as agent) to load the ssh keyfile
        AddKeysToAgent yes
        # For ssh to then use Keychain as an agent
        UseKeychain yes

~~~
ansible
You can use the keychain to store keys, but if the key is ever saved in the
old (broken) format, you've got a problem.

------
ejholmes
If you have a recent MacBook, use
[https://github.com/ntrippar/sekey](https://github.com/ntrippar/sekey) for SSH
keys. Private key is stored in the built-in HSM (Secure Enclave) and access is
controlled by biometrics (TouchID). You can do the same with a Yubikey, but
it’s not quite as good because a) you can lose your Yubikey easily and b)
there’s no biometrics.

Private keys stored on filesystems is an antipattern.

~~~
Whitestrake
Ahh, if only Apple released a non-touchbar MBP with Touch ID! I'd have used
this in a heartbeat.

------
amenghra
My ssh key is stored in a yubikey. Haven’t had to type a password for ssh
purpose in a long time!

~~~
anilakar
A TPM chip is also easy to use with libsimple-tpm-pk11. Not a perfect solution
but at least the private key cannot be stolen and misused without direct
access to the computer.

------
knorker
If you have really old clients you may apparently be able to do this instead:

[https://martin.kleppmann.com/2013/05/24/improving-
security-o...](https://martin.kleppmann.com/2013/05/24/improving-security-of-
ssh-private-keys.html)

------
bogomipz
>"You might ask yourself how OpenSSH ended up with this. The sad answer is the
OpenSSL command line tool had it as a default, and now we’re stuck with it."

Could someone elaborate on this? Is this saying that the ssh-keygen utility
shells out to the openssl-command line tool which at one point defaulted to
md5 for encryption? If this is the case why would be stuck with it?

~~~
lvh
Backwards compatibility. I don’t think it shelled out, but close enough.

------
spiznnx
It's best to assume all passwords need to have sufficient entropy to withstand
a brute-force attack on some super fast hash until proven otherwise.

I didn't change my macOS password to 8-characters (to save myself 20 seconds a
day) until I thoroughly researched what key-stretching was being used under
the hood.

------
bogomipz
The author states:

>"Finally, most startups should consider not having long-held SSH keys,
instead using temporary credentials issued by an SSH CA, ideally gated on
SSO."

Can someone explain what "gating" SSH CA issued key with SSO means? How would
SSO "gate" the keys?

~~~
lvh
You use SSO with strong authentication to sign a temporary credential for SSH.

~~~
bogomipz
Thanks, might you know which SSO implementations support this?

~~~
lvh
I like Teleport. It's not an SSO implementation but if you pay Gravitational
you get SAML. BLESS, the Netflix solution, relies on you being able to call a
Lambda. You can SSO that as well via STS' AssumeRoleWithSAML. (But be careful,
AWS SSO should be like the last SSO you implement, generally hardening IAM is
super complicated, talk to me for more details. We owe y'all a bunch of dinky
"here's how you do IAM" blog posts.)

EDIT: I previously incorrectly claimed that you need to pay Gravitational for
Teleport SSO. That is incorrect: you only need to pay for _SAML_, specifically
-- I forgot that you can GitHub auth into it, which is a form of SSO. (Though
for most of our customers I think that single trust store is a core part of
SSO, and GitHub isn't a good SSO by that metric, by vritue of account reuse
and the fairly tenuous links between users and organizations. GitHub does a
great job of modeling open source interactions, but that model falls over a
bit when translated to commercial software engineering orgs.)

~~~
old-gregg
The open source edition of Teleport has SSO support for logging via Github
[1]. The enterprise version adds enterprise SAML implementations like Okta,
Auth0 and things like ADFS.

[1] [https://gravitational.com/blog/replace-static-ssh-keys-
with-...](https://gravitational.com/blog/replace-static-ssh-keys-with-github-
oauth/)

~~~
lvh
Whoops; my bad!

------
valenciarose
Use SSH keys generated and stored on some sort of PKCS11/PIV hardware device.
It’s relatively easy to do now. Don’t trust a general purpose computer to keep
your keys secure.

Better yet, use U2F where you have control of how you are authenticated.

------
technion

        It’s tricky to find out what this DEK-Info stuff means
    

This is really where the modern crypto protocols and libraries are doing well
imo. Even if something that opaque is implemented perfectly, it presents a
risk.

------
nebulous1
tl;dr to move to the newer more secure format use

    
    
      ssh-keygen -o -p -f file_name
    

such as:

    
    
      ssh-keygen -o -p -f ~/.ssh/id_rsa
    

not necessary for ed25519 as it can only be stored in the new format.

------
lwf
Filed [https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=905407](https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=905407) to change the Debian default.

------
otterley
Is anyone aware of a successful infiltration of a system that occurred by
stealing an encrypted SSH key with a reasonably good password and successfully
decrypting it?

~~~
pferde
If such an infiltration is successful enough, nobody (besides the attacker)
would be aware of it. :)

~~~
otterley
It's the damage we'd notice -- exfiltration of data, malicious destruction,
etc. The intrusion is but a step towards the real goal.

~~~
pferde
The damage wouldn't necessarily be noticed for what it is. What if such
exploit was/is used to get more compromised zombie boxes into botnets for hire
(such as the one currently spamming Freenode and other IRC networks for the
past week or so) ?

------
digitalnalogika
What are you thoughts on using Krypton? [https://krypt.co](https://krypt.co)

------
w8rbt
All my ssh key passwords are generated from password managers. They are random
and strong.

------
gigatexal
What if you generate the public and private keys without a password but keep
it blank?

------
adultSwim
I'm finally vindicated for not using a password on my key :P

------
drb91
Does this affect non-RSA key types too? I use ecsda25519.

~~~
progval
> How do you fix this? OpenSSH has a new key format that you should use. “New”
> means 2013. This format uses bcrypt_pbkdf, which is essentially bcrypt with
> fixed difficulty, operated in a PBKDF2 construction. Conveniently, you
> always get the new format when generating Ed25519 keys, because the old SSH
> key format doesn’t support newer key types.

------
nerdponx
So is that what happened? Somebody got a hold of one of the developers'
private SSH keys, and cracked the encryption?

------
exabrial
Seems like gpg auth would be a much better choice the ssh keys

------
jwildeboer
Nice deflection of the real madness here that a compromised npm package gets
to read stuff in your user directory.

~~~
annabellish
I mean, of course it does. Most people run their development environments as
their own user, and most package management systems have post-install script
support.

Installing random software from the internet is problematic.

~~~
vemv
I'd be interested in running my dev environment with a dedicated user account
-with no access to the main account's personal files, credentials, etc-, do
you know of any guides for doing so?

(I could just give it a shot, but I reckon there will be gotchas)

~~~
angry_octet
What platform? GUI or shell?

Docker/LXD or a dev VM lets you access both simultaneously. Or create a
separate user, but not as secure or convenient.

~~~
vemv
macOS. Not much interested in VMs, for efficiency.

Yeah I was playing with a separate user account, could get some basics working
but I wonder how far could I get with that.

What are the bigger security risks for that approach? Assuming constrained
file permissions, and that no secrets are in ENV
([https://gist.github.com/telent/9742059](https://gist.github.com/telent/9742059)
)

~~~
angry_octet
A normal user has full access to the kernel API. Always kernel info leaks,
occasional easy exploits.

On macos you can use Apple's native sandboxing. See for example
[http://mybyways.com/blog/creating-a-macos-sandbox-to-run-
kod...](http://mybyways.com/blog/creating-a-macos-sandbox-to-run-kodi)

------
mrrandomguy11
Use key AND interactive password. This way attacker will have limited rate of
password guessing and you will now from logs that your key is compromised.
Your key will be used only for first step of authorization. You can slap
additional password for that key but it is not even needed. Additionally from
what I remember Ed25519 keys are stored by default in the new format.

~~~
lvh
Don't do this. Get an SSH CA and temporary credentials, and make your
authentication to that SSH CA actually good (e.g. SAML with mandatory U2F).

~~~
Boulth
What's the difference between SAML+U2F vs smartcard stored SSH key? Both
cannot be easily stolen or abused. Additionally Yubikey have touch-to-use
policy that further protects the key usage.

