Improving the security of your SSH private key files 330 points by martinkl on May 24, 2013 | hide | past | web | favorite | 101 comments

 I keep my SSH key pairs on a smart card. It is very cheap to purchase a smart card and card reader (~£25 for a gemalto reader & a gemalto .Net smardcard) and if you're buying in medium to large quantities for a business the cost is even less.This has some major security and convenience advantages over keeping keys in a file. Firstly, you can generate the keypair actually on the card so that the only device that ever has, and will ever have, the private key is the smartcard. Secondly, the key is secured by a pin and, by default, it will block the card after 3 incorrect pin attempts and after 3 incorrect attempts to unblock the card it will permanently erase the secure storage on the card. Also you can easily make use of your keypair on multiple computers, even untrusted machines, without compromising security. I keep my smartcard in my wallet, so I always have access to it where ever I go.Alternatively, if you don't want an actual card you can get smartcard-like devices are are physically similar to a USB stick.
 > it will block the card after 3 incorrect pin attempts and after 3 incorrect attempts to unblock the card it will permanently erase the secure storage on the card.Why three? Can someone actually guess your pin in 4 tries? I hate the number 3 used for these things. Raise it to 20 and you'll have virtually the same amount of security, but the false positives will drop to 0.
 You can change it to whatever you want. Here is the configuration options for my card for the user pin:http://i.imgur.com/hHxlh2f.pngThe whole thing is even more complex that that. You can actually have different pins for different roles. For example, you could setup a different pin that can only be used for authentication with a specific keypair/certificate. Also a pin does not even have to be a pin as such. The admin pin on the card (i.e the role that can unblock the user pin) is actually a challenge-response as it is designed so that a user can phone up IT support or something and tell them the challenge from the card and IT support can give them a response that will then unblock their normal pin.
 If someone gets hold if your smart card, they can enter a wrong pass phrase to remove your private key, right? So, you have to have multiple of these smart cards (that all share the same private key), some in your wallet, some in your house and some in a bank or vault somewhere. Otherwise, you can lose access to your private keys forever.Am I right, or I'm missing something?
 Depending on the cards and their configuration, you could generate the key outside (say, on a trusted machine at home) and upload it to the card. If the cards gets stolen, and you trust the security of the card to not leak the key, you could just get yourself a new one and upload the old key again.The other philosophy is to generate the key on the RSA card, and never allow the card to export it. Then you consider the private key dispensable and will have to update all systems to accept a new card's private key.
 update all systems to accept a new card's private key.That sort of a chicken/egg problem though, right? It assumes you've got console or some other means of access that will take another credential.
 I don't see a situation where you are unable to get console access as an acceptable situation. You don't need to be using a smart card to require local console access if things go bad. What about if the SSH server fails or someone applies a bad firewall rule that prevents SSH access?
 You don't need multiple cards with the same private key. You can use multiple cards with different private keys, that are all authorized for your server(s). If you lose one card, just remove that card's public key from the authorized keys.
 It does not matter if you lose your SSH private keys though as they are disposable and you can just generate new ones, it's not like they are being used to encrypt anything. To add your new public key to your servers you either get someone else who can SSH in and su to root to do it for you or you login locally (via kvm or similar). Larger deployments most likely have some centrally managed system for dealing with SSH keys.
 I'm quite interested in this, as I have some old smartcard readers around from the days of Nagra/Nagra2. Care to share any more details?
 The smartcard needs to support openpgp smart cards. You then create an RSA gpg key with three subkeys - signing, encryption, and authentication. You'll have to enable the advanced key-creation mode for the last one. That authentication subkey then becomes your ssh key.There are plenty of tutorials online, but none are particularly good. I've been meaning to regenerate my keys, so maybe I'll take notes and try to write up a good one.Here are a few links:The last one is interesting - it's a smart card reader and integrated smart card built into a usb stick.
 Most likely will depend on what smartcard you get but here is how I do it with my Gemalto card:I use the Minidriver manager tool from: http://www.gemalto.com/products/dotnet_card/resources/develo... although you could use something like pkcs11-tool.Generating a RSA keypair is as simple as right clicking on an empty container, selecting OBKG Container and filling this out:http://i.imgur.com/uMY53OD.pngIt will display the public key but it is in hex and uses Microsoft's PublicKeyBlob structure. OpenSSL will convert this to PEM for you though, which is what you want for SSH authentication. Something like: openssl rsa -pubin -inform MSPUBLICKEYBLOB -in "C:\path\to\my\publickey" -outform PEM -out "C:\path\to\my\publickey.pem"  Open this new file in a text editor and get rid of the begin and end public key lines and just smack it all on one line with "ssh-rsa " (note the space) in front of it. You can then add this to your server's authorized_keys file.OpenSSH can use a pkcs11 library for authentication (-I option), which is also avaliable from Gemalto's website (these are generally specific to the smartcard), and on Windows there is a version of PuTTY called PuTTY SC that will also let you use the pkcs11 library.
 I have done something similar -- I've run ssh-agent on my phone, then I ssh from my phone to my PC with the agent-forward (-A) option so that any ssh connections I do from my PC forwards the authentication requests to my phone. (This was with my Nokia N900 phone -- I'm working on setting up a similar thing for Android).
 Of course, this doesn't address what I consider the greatest weakness in using ssh - distribution of host keys.By design, ssh does not make use of certificates. So there is no way for ssh to know upon connecting to a host for the first time whether the public key presented by that host is authentic. After the first connection, ssh caches the keys in ~/.ssh/known_hosts, and will then give you a big warning if the key changes. I imagine when this happens many folks blindly delete the the cached key and re-connect.So you should be aware of the potential for MITM attacks to occur unless you have some out-of-band mechanism for distributing or authenticating host keys.Edit: huh, apparently ssh added support for certificates. "ssh-keygen supports signing of keys to produce certificates that may be used for user or host authentication." So now you just need to distribute your CA certificate everywhere and you're golden. http://justanothergeek.chdir.org//2011/07/howto-authenticate...
 So now you just need to distribute your CA certificate everywhereAs opposed to writing down the key signature of your server's host keys on a card in your wallet? You only need to look at it the first time you connect to the server, and it's a lot less of a pain in the ass than trying to install custom CA certs on every machine you SSH from.
 If you have one server, that sounds great. If you have 60, it does not.
 For the love of god why have 60 servers with ssh available open to the world? Ssh through trusted gateway machine! Then you need a single signature you trust and you can firewall off the rest of your hardware.
 For the love of god why would you assume this has anything to do with "open to the world"? When I proxy through the trusted ssh gateway to the firewalled servers (using ssh ProxyCommand), I still like my client to verify the signature.
 I would assume that if your internal network is "trusted", then the risk of a MitM is low enough that you wouldn't have to check the signature. I agree that an extra layer of validation is always a nice thing to have, but everyone has different levels of acceptable inconvenience with regards to security.
 You could store the signatures on one server, and import them in bulk. Then you still just need to write down the one signature.
 Yes, there are many ways you can work around the problem of ssh needing manual inspection of individual host keys, and the method you propose is pretty reasonable. But building the verification into the infrastructure is a very sensible and nice way of handling it, and lets everyone throw out their individual ad-hoc solutions to this problem.
 Or if you're running windows you can use group policy and windows PKI to get it done automatically. But I haven't found out how to do the same thing under a nix environment.
 If you have 60 servers you are hopefully using some kind of configuration automation, like puppet or chef...?
 Yes, but in my experience, doing that for client machines (where you need to check the fingerprints) is pretty atypical.
 You can put SSHFP records in the DNS. See "VerifyHostKeyDNS" in the SSH man page. You'll also want DNSSEC set up of course.
 After you get a validating stub resolver, since your OS probably doesn't ship with one by default.
 Doesn't take long to do an "apt-get install unbound" and then modify your network settings to use 127.0.0.1 as your resolver.
 And if you're using a DNS service that doesn't support SSHFP records, you can generate and distribute a base known_hosts file with your favorite configuration management solution.
 While many suggestions are correct, i also like the monkey-sphere one because its based on GPG trust model.With the CA you have to have a base trust in a CA cert, with GPG you trust whatever you want.
 Even though MITM is a very important attack vector, we've seen recently how the strength of the private key file encryption can overnight come under scrutiny of world-wide IT media (spoiler: Linode private key has a 'secure' password):
 > This ensures the key is never transferred somewhere it can be copied.But it does not ensure that the computer will add another entry to the authorized_keys file in the host as soon as you access it. Or install a rootkit there with another kind of backdoor.If you don't trust your computer, don't give it access to your servers.
 This is why I haven't bothered with smartcards. If you login from a compromised desktop, it's pretty much over, even with a smartcard.There is no real way around the evil maid attack. If I login from a compromised computer, if that computer is compromised by a competent attacker, it's over.The advantage of a smartcard, if and only if you've got a smartcard that requires some action to authenticate (like a physical pin pad) is that only the servers I've logged into while compromised are compromised (assuming you don't have some kind of jelly doughnut setup where all servers can login to all other servers) whereas if I have an encrypted private key, well, if I decrypt it while the attacker is on the box, the attacker has compromised /all/ servers I have a login on.
 I'm not sure if I understand what you're saying, or if _you_ understand what you're talking about to be honest.. ;-)If you need to access a server you'll always need a computer. authorized_keys are public keys only. it does not matter if other computers have your public key. all it does is give you access.the private key of your ssh key(s) give access to several servers, thats why its the part that you want to protect. if one server has a rootkit, well, that sucks. but if that rootkited server can access all the servers YOU can access, you're screwed.
 He's saying a latent program could hijack your established ssh connection to add another public key corresponding with an attackers private key to get long term access.
 > Making your passphrase twice harder to crack is nice.. but whats 15 days instead of 7? Making your passphrase 10000x harder to crack would also be niceYou mean like doing 10000x iterations of md5? Btw, it's not even 15 days instead of 7 any more. You need to measure it purely in money. EC2 cost is nodes * time, so if you need to double the time it is just doubling the cost.
 What prevents a compromised laptop from reading your private key in the clear from your smartcard? Even if you need a fingerprint read or PIN, I mean at some point you need to get the decrypted key read from the card so that it can be used.
 The private key is not accessible outside of the smart card. All the cryptographic operations that require the private key are done by the smart card itself. Cryptographic smart cards are not just a dumb storage device, they are basically a computer. They have a CPU (my card has a 66Mhz 32bit RISC CPU) and normally run some simple OS. You can actually write applications that can run on the smart card using, for example, a subset of Java or the .net framework.Edit: For example, you can view what is currently running on the smartcard: http://i.imgur.com/Pgwz4la.png or view the filesystem on the smartcard: http://imgur.com/aLsV6pf
 Ah, I guess that's why they are called "smart". Thanks, learned something.
 You may be interested to know that RFID cards are also 'smart', although less so. They have hardware circuits that can perform some cryptographic routines, and receive enough power to perform many calculations. So an RFID reader will send a challenge, the card will perform some operations based on a secret key, and send back the result. The reader is connected to a central database, that also knows this secret, so it knows if the response is valid. As such you can't just 'copy' an RFID card, or even replay a valid request (since the challenge is unique).
 This is straying out of my knowledge, but I'm pretty sure most smart cards perform the encryption on the card itself. The public key can be exported, but without specialized hardware and/or software (that you hopefully don't run all the time in an unlocked fashion), you can't extract/set the private key.
 > openssl pkcs8 -topk8 -v2 des3 \> -in test_rsa_key.old -passin 'pass:super secret passphrase' \> -out test_rsa_key -passout 'pass:super secret passphrase'Is there a way to do this without including the password on the command line? Because somehow I don't think having your private key's password in your .bash_history will improve the security of your key. Oh I see, just don't specify and it'll ask. I get that he's doing this with a sample key, but you just know someone's going to scan through this and do that command on their real key.
 If you put 'histcontrol=ignorespace' or 'histcontrol=ignoredups' in your ~/.bashrc file it will not record commands that begin with a space [1]. I believe that 'ignoreboth' is the default on many Linux distros.
 You still shouldn't enter these kinds of commands on a multi-user machine, because anyone running 'ps' will see the commandline arguments.
 Leave out -passin and -passout.Provide your current passphrase at the first prompt ("Enter passphrase for :") and your new passphrase at the second and third prompts ("Enter Encryption Password:" and "Verifying - Enter Encryption Password:"). You may use the same or a different passphrase on your new key.
 To avoid a command showing up in history, start the whole command with a space and the line won't be recorded to history. Or if that horse has already bolted, you can use the 'history' command to find the line number and then delete it.Of course, most things that require passwords will request that you manually enter one if you leave the arg off.
 He did in the last example that shows how to do this on an actual file ...
 To clarify, my point is that a blog post about improving your security should not have an easily cut-and-pasteable reference for substantially decreasing your security, even if it's intended to be used on a dummy file.
 $export HISTFILE=/dev/null  You want:unset HISTFILE  Don't do that in your .bash_profile unless you run login shells on all of your terminals!Setting HISTFILE propagates that setting to child shells - unsetting HISFILE does not.  Undefined = Default  I couldn't help but notice that the original SSH key was encrypted with AES-128-CBC and the "more secure" one mentions 3DES in the ASN.1 structure. This makes me question if the "improved" SSH key really is more secure.I'm not an encryption expert. Can someone else weigh in on this?  I believe you can tweak the algorithm to suit your requirements, e.g. openssl pkcs8 -topk8 -v2 aes-128-cbc -in ~/.ssh/id_rsa -out test_rsa_pkcs8  As far as the strength of AES vs. 3DES, the difference is probably negligible for most practical purposes. AES is however the newer standard, and is more flexible, performs better on different hardware etc, so seems like an overall better choice, but I would say not so much for any practical security properties. (i.e. both are pretty strong).  You can get a list of the available cipher commands with openssl list-cipher-commands  A description is available if you do man enc  Doesn't seem to be any reason you couldn't use AES-256-CBC if you wanted to. Not a security expert though, so other than speed concerns, I don't know the repercussions of doing so.  The reason 3DES is avoided is because it is significantly slower than AES and usually leads to generally poorer performance. Security-wise, it's fine to use if you're willing to make these performance tradeoffs (eg, you might want it for backwards compatibility with some ancient DES-friendly system).  The fact that 3DES is slower is actually an advantage if you are trying to prevent against brute-force attacks. Given that we are encrypting binary data, it's unlikely that knowledge of the plaintext (for example, knowing that it's probably US-ASCII and so the 8th bit is probably always zero) is not going to be available to the attacker. Other potential attacks, such as related plaintext attacks, will also not be available to the attacker. Given that brute force attacks are the most likely threat scenario, using a 3DES which is slower is a reasonable choice.  Double-negative in your second sentence, and I think it's not what you meant. I wouldn't be so pedantic, except that crypto is worth being pedantic about.  Thanks for the replies. I'm certainly no expert on this stuff. A few years ago, I implemented DES as a learning exercise [1]. I recall reading something at the time about security concerns with DES. That was several years ago, so I'm probably mis-remembering. Wikipedia vaguely refers to concerns about security being a motivation for AES, but doesn't explain what those concerns were [2].  Key and block size were the concerns. Key size specifically. Cryptographically speaking, DES has been incredibly good and resisted pretty much all attacks but the key size has made brute force feasible.   ssh-keygen -t rsa -N 'boobooboo' -f test_rsa_key history ... 964 ssh-keygen -t rsa -N 'boobooboo' -f test_rsa_key  Best if you let commands prompt you for a password, rather than putting them on the command line for ps, history and everyone else to see.  Tip: you can prepend a space at the beginning of the command and it won't be saved in the history (though I agree that letting the command prompt for a password is better)  Excellent. Although that can be set to not do that. I've explicitly set it to "ignoreboth" for so long that I'd forgotten why, so it seems new to me today. :)From man bash: HISTCONTROL A colon-separated list of values controlling how commands are saved on the history list. If the list of values includes ignorespace, lines which begin with a space character are not saved in the history list. A value of ignoredups causes lines matching the previous history entry to not be saved. A value of ignoreboth is shorthand for ignorespace and ignoredups. A value of erasedups causes all previous lines matching the current line to be removed from the history list before that line is saved. Any value not in the above list is ignored. If HISTCONTROL is unset, or does not include a valid value, all lines read by the shell parser are saved on the history list, subject to the value of HISTIGNORE. The second and subsequent lines of a multi-line compound command are not tested, and are added to the history regardless of the value of HISTCONTROL.  i'm not sure if it works in all shells, but at least in zsh/bash doing$ echo "this will not be saved in history"  as opposed to $echo "this will be saved in history"  can be useful  AFAIK, in bash/zsh you have to do export HISTCONTROL=ignoredups:ignorespace  in .bashrc or .bash_profile for the above trick to work.fish, OTOH, seems to do this by default.  That, or use read: read password > secret-password-followed-by-enter-here sensitive-command --password$password unset password  edit: formatting
 It provides some security against attackers that aren't capable of subtlety. If yank your unencrypted hard drive out and walk away with it in my pocket I can't use your encrypted SSH key to impersonate you to other machines unless I can break the passphrase.
 If your HDD is stolen they can plug it into another computer and copy any file they want.
 They would get quite a few pseudorandom numbers of mine, aside from a self-compiled Linux kernel. :-)
 > aside from a self-compiled Linux kernel.Likely sitting on an unencrypted /boot, waiting to be replaced by one with a keylogger, am I right? Of course a sane person doesn't trust a system that's been compromised before nuking it and reloading from a clean image.
 There is unfortunately relatively little you can do to thwart such an attack, apart from keeping your notebook with you at all/most times.Though using a USB key for /boot might be an idea, it is a little less clunky than a ThinkPad and since I suspend to RAM most of the time, it could even be practical. Hm.
 Can't TPM be used for this? It could verify your /boot with keys external to the disk itself. I'm not sure if somebody has actually built a solution that uses it yet.
 Sure it can, as can its evolution in the form of UEFI's Secure Boot, the problem is everyone wants to label these as technologies to enable lock-in instead of technologies to provide a trusted boot chain to ensure your system isn't compromised.
 It could (the same goes for Secure Boot, in some sense). But the three-letter agencies from whom you want to protect yourself here likely have backdoor keys.
 Creating multiple SSH keys is an interesting form of obfuscation, but it relies on trimming your bash command history and not using ssh-config files.
 No, I think more like one of three things:a) just your laptop, b) your laptop and all files, or c) your laptop and all your files, and access to servers as per above.I like "a" the best, and it's achieved through full-disk encryption. You don't even need a new passphrase for that, considering that you probably already password-protect your computer.
 There's Crypto-Stick [0], a security USB key with many interesting features. It's essentially an OpenPGP card with some added features.What excites me about their project is a very simple file-system based interface they are planning to implement for their upcoming version. Plug the key in, it looks like a regular USB key, put a file in a specific place, get an encrypted/signed/whatever file from the file system from another place. No driver or software required. I hope their project takes off and people start buying these keys. The price is a little high right now (59,00 €) and it is currently not available.
 This is interesting - I've never thought to investigate how ssh stores keys.For managing passphrases, I strongly recommend the program keychain:http://www.funtoo.org/KeychainIt's a thin wrapper around ssh-agent, but ultimately means I only need to enter my passphrase once.The nifty feature is that you can add a command to your .login to clear your passphrase, so it's safe to use on remote hosts - if someone else is able to ssh in, they only get access to that host, not others that your ssh key allows access to.
 I just recently started using keychain as well. It's worth noting that it also supports gpg-agent.
 It is possible to store SSH private keys in gpg-agent, which uses the openpgp-s2k3-sha1-aes-cbc algo to encrypt the key for storage.s2k3 is an iterated-and-salted s2k algorithm from RFC2440, the protected private key format is: https://gitorious.org/gnupg-org/gnupg/blobs/master/agent/key...The number of iterations is at least 65536, whereas with pkcs8 openssl uses 2048 iterations by default.More details on how to use SSH keys with GPG agent: http://budts.be/weblog/2012/08/ssh-authentication-with-your-...
 I believe in having unique keys for each client device, then putting them in a hashed authorized_hosts that you can treat as a unit.That way, if someone steals my (FDE, managed) laptop or phone, I can revoke just the credentials on that client device, not losing access to all my servers.Using tamper-resistant storage for the keys is a great idea; I wonder if anyone has done TPM for ssh on windows, or just trousers on linux. Sadly for Macs the only option is some kind of external USB token :( I kind of wish someone made a bluetooth 4.0 le device which could hold keys and be used as a smartcard/key device, maybe with trusted user I/O too. Maybe in a watch form factor.
 A crypto puzzle: Is it safe to encrypt several keys with the same passphrase? Or is it possible to reveal the keys if the same passphrase was used to encrypt them?
 The key is encrypted using the IV as a salt and your passphrase. So even if you encrypt the same key many times, the encrypted data will look completely different unless the iv is the same.
 I think that this is not a risk (between the PBKDF2 salt and the IV of the underlying cipher). IANAprofessionalcryptographer, however.
 To be fair the DEK-Info IV should be different for each key generated, so rainbow tables and other common precomputation attacks are pretty much out of the question.It's not as good as PBKDF2, but it's better than nothing and is probably why stretching isn't used.As for the argument about being susceptible to a dictionary attack, well if you go to the trouble of using key-based auth then use a dictionary word you're kind of asking for it really.
 > To be fair the DEK-Info IV should be different for each key generated, so rainbow tables and other common precomputation attacks are pretty much out of the question.That's valid; however, when you can compute 33.1B MD5 hashes a second, who needs rainbow tables? http://blog.zorinaq.com/?e=43Six and seven digit passphrases are easily brute-forced.
 My private SSH keys have passphrases of 19-20 random characters. I store them in a KeePass database (AES encrypted) so that I can copy/paste. For keys on my Mac, I've also allowed the KeyChain (Triple DES encrypted) to remember it so that I don't have to copy/paste it every time.I think this approach should be more secure than trying to set memorable passphrases for all my keys. Thoughts?
 Where do you store the passphrase of your passphrase database ?
 I don't know about the grand parent, but personally I store that one in my brain. It's about 32 (maybe more?) characters and quite complex, but it's the only passphrase I need to use regularly so it's etched into my mind.
 You don't even need to have a pass phrase that long with KeePass. It allows you to set the number of rounds of encryption to perform to make brute forcing even short master passwords infeasible.Of course, that assumes the master password is chosen such that it isn't susceptible to your typical dictionary/hybrid-dictionary attacks.
 You might still want to write that one down somewhere and store it somewhere safe (and probably hard to get to). A friend of mine recently forgot his PIN. A four-number code used regularly for over ten years. Just gone like that.Your memory is a SPOF, and while generally robust for this sort of thing, it's not bulletproof.
 Same here. I also require a key file for KeePass. It's not encrypted, but it's just one more thing a bad guy would have to acquire to get in.
 I'm using YubiKEY USB token for the passphrase (it acts as a keyboard driver and token has a button to press to enter it). Also has support for OATH.
 You could use challenge response with the yubi (or just oath) and save the ssh key passphrase in keychain.. or you could get a yubikey neo and store the ssh key in the yubikeyneos can store the rsa key and use it to authenticate via ssh, as it has opengpg smartcard emulation support
 Hey, if you're going to remove the old key, remove it securely: use srm!
 As far as I can tell, this only works with DSA and RSA, but not ECC keys (~/.ssh/id_ecdsa). Thoughts?
 Ah. I am an id10t. Helps if I type the command correctly. Carry on.
 I use a 4096 bit length rsa key with a 200 something bit password. Am I safe?
 Or just use a proper OpenPGP card.

Search: