Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
A sane SSH(1) key management example (try.popho.be)
77 points by moviuro on Aug 18, 2022 | hide | past | favorite | 64 comments



> How would you look at someone using a single key for their car, house, safe, work place, and so on?

This is a terrible analogy, and I don't see any other justification for this setup.

An SSH "key" is also referred to as an "identity". Contrary to a car key, it is not tied to the car, but to the client's identity. It is more like a badge than a brass key.

A car key has the limitation that your lock can only accept one key, and can't be easily updated to accept/refuse keys. You also can't remotely give somebody access (you have to hand them the key) and you can't prevent someone with the key from making a copy without returning it.

An SSH key pair has none of those limitations so it's really not clear why anyone might want to manage them like car keys. If anything, the root password is more like a car key than a key pair is.


It's a great analogy, for many people.

The distinction between treating SSH keys more like keys, or more like a badge, is heavily milieu dependent.


SSH public keys are called public keys because they're totally safe to distribute anywhere. That's rather the entire point of asymmetric cryptography.

So I guess I don't really understand the problem being solved here.

And yes, I would very much love to have a single key that worked for my house, car, safe deposit box, etc. One that isn't my smart phone, of course.


Privacy. If someone owns two servers you're logging into, they can associate your logins if you use the same key. Another case is associating you with your GitHub account (as your ssh keys there are public)


This is fingerprinting across different identities which is a form of discovery. It’s why I have separate ssh keys for work and personal projects in the first place to better avoid associations across different online identities (with the full understanding that if I’m targeted I’m already against attack vectors that require much more sophisticated defenses). Firstly, for work projects I can give up my private key to my employer as I leave and I won’t care because it was only ever used for work and it would be some reassurance to my colleagues that they can at least access the stuff I used to have access to.


If you run a dark net market and your servers are at the risk of being seized, then no, you don’t want stuff like whoami.filippo.io to work.

It all depends on your threat model, I guess.


Assuming you use different keys for different services, the answer would be: privacy.


My googling skills are failing me. Can anyone help explain what the "(1)" after SSH means? I guess I always thought it was a footnote marker or something (mainly I just ignored it when I saw it) but I'd love to know what it means.


The (1) means Unix manual pages, section 1.

The "ssh(1)" means "read about ssh in man page section 1".

Section 1 is for general commands.

https://en.wikipedia.org/wiki/Man_page

Pages are traditionally referred to using the notation "name(section)": for example, ftp(1). The section refers to different ways the topic might be referenced.

    1 General commands
    2 System calls
    3 Library functions
    4 Special files such as devices /dev
    5 File formats and conventions
    6 Games and screensavers
    7 Miscellaneous
    8 System administration commands and daemons


In this case, running "man 1 ssh" is likely the same as "man ssh" (ssh only has a man page in section 1). Many systems will have separate pages for for example "man 1 crontab" and "man 5 crontab".


On Macs, Secretive [0] is great. It creates keys in the secret enclave, from where they can't be read, only used for signing requests. TouchID authorisation is optional but it's so quick and easy that I keep it on for all keys.

It can also use Smart Cards (Yubikeys are called out by name in the readme).

A forwarded agent will have the same level of security, meaning that if the forwarded agent needs to use a key in Secretive, it will have to be authorised locally - and even if TouchID is disabled, you are notified if a key is used.

[0] https://github.com/maxgoedjen/secretive/


OpenSSH since 8.2p1 supports FIDO2 U2F keys directly (via libfido) using no proprietary Yubikey functionality, as the new ed25519-sk key type (sk for security key). The server also needs to be 8.2+ but doesn’t need to be compiled with libfido.


That's cool, but can it use the secret enclave and TouchID?


All but one of my Macs lack them. I also need a solution that works on Linux.


I was asking about using the secure enclave and Touch ID via the direct support on OpenSSH.

As for cross-platform compat, I wonder if you can use the same keys on the Yubikey via both Secretive and the native OpenSSH support. If it does I might look again into getting a Yubikey.


> It’s common knowledge that you shouldn’t put all your eggs in the same basket, but most of the time on IRC or on reddit (or the Internet at large, really), I see people using one single ssh key for all uses. How would you look at someone using a single key for their car, house, safe, work place, and so on?

a bit envious tbh. I would be so annoyed if we would get a key fob for every door at work instead of one key fob that can be reprogrammed.

The comparison drawn here is bad because you would look at the person funny because "How the hell is someone able to replace every lock so that this works?"

At work we have a script that pulls all our gitlab keys and adds them to the authorized_keys section if you should have access to a server. In what scenario is leaking your identity over ssh really a problem? If I want to connect to a server normally I either own or administrate the server. Next use case is a leaked private key.... How the hell do you leak your private key? There are 3 scenarios I can think of:

  * you copy it to a host/usb drive and someone else has access to it
  * a new attack is found to generate a private key to your public key
  * someone gets access to your machine and steals it
1) you shouldn't do 2) leaves all your keys vulnerable 3) every key on the machine needs to be replaced

> As an added benefit now, if one of your ssh keys ever leaks, there’s only one place to remove it from ~/.ssh/authorized_keys (where the login@hostname comment is still present).

Do you do this for every machine you own? Sounds like a real pain. Maybe only because of my setup with passwords and keepassxc.

If you fear leaking your key maybe a fido2 device and/or password would be a better solution. Don't get me wrong I too have more than one ssh key, but this seems overly excessive. Since this solutions looks rather clean it maybe isn't, but I don't see many advantages here. But it is a nice setup nonetheless. I could reasonably easy implement something like this on top of my existing setup, but right now it seems only to add more administrative work. Especially since I really like the idea of an asymmetric key that opens my doors. The only downside for ssh keys is that you can't invalidate them in a central location.


Yeah, I would entirely love it if I could use the same key for everything I have that takes a key. That'd be great. As it is I just don't bother with keys for, like, back doors to my house and such. They're in a drawer somewhere and never get used. I just can't enter those doors from outside, if they're locked.

One key (well, I mean, I'd want copies of it) for everything would be excellent.

Terrible analogy because I'd look at them like "damn, they've got it figured out!"

[EDIT] Thinking further, the only way this falls apart is if I want to give someone else a key to just one thing, but that's a non-issue with ssh, so the analogy is still comically backwards. "But what if someone gets ahold of one key! Same key for everything, that's access to all your stuff!" well shit man, that's about the same as getting one of them if they're all different, in most cases—where do you think I keep my extra keys? And the couple I carry, so are likely to get in someone else's hands by accident or whatever, are the "give you the kingdom" type anyway. Get my car key, you can get in my house (garage door opener). Get any house key, you can probably get in my car, plus the rest of my keys (e.g. safe) are in there because I don't carry them around all the time.

Yeah, letting someone borrow a key is the only time this might be inconvenient, and again, that's not a factor with SSH.


You know, there are master key systems that do exactly this: you can have a master key that opens everything and special keys for just one of the locks. There are even grandmaster systems with some locks opening subsets of locks. I'm personally a huge fan of a certain Finnish manufacturer but there are many others. You can't usually put one of those compatible cylinders into you car, usually, and there are other limitations, but, still, even just for doors and lockers it's a great thing to have. Also, usually, those expensive cylinders are reprogrammable in case of a compromise and there are other security features (keys being impossible to copy is the one I like most).


>How would you look at someone using a single key for their car, house, safe, work place, and so on?

One key to rule them all? Honestly, I would be jealous.


Sure, but as soon as you've got to leave the car key at the shop for the car inspection, you might rethink that approach.


If you can generate infinite keys for your car, invalidate old ones and limit the privileges of each key you may think differently. I would love to give the car mechanic a key that only works to drive around the lot or test if it starts.


> invalidate old ones

This one is non-trivial in SSH land (unless you go with a CA approach, of course). Lots of authorized_keys files strewn about everywhere...


If you distribute a key revocation list to your servers, it will override anything in anyone's authorized_keys file (on that server of course).

Even with a CA approach (unless the expirations are sufficiently short) you will need to do something like this.


Yes, but if you're distributing revocation lists you might as well distribute new authorized_keys files?

Or wait, maybe I'm misremembering... does sshd support querying remote revocation lists? If so, point conceded. You still have to have to worry about the scenario of DoS or similar preventing it from fetching new lists. (I don't trust revocation beyond as being a QoI thing.)

I do favor the CA approach with e.g. 24hr expiry just because it's 'fail safely by default'. Of course one should ideally use much more frequent renewal, but OpenSSH has its limits wrt. that. (Kerberos seems saner in terms of results, just not in practice because it is/was so obnoxious to set up.)


> does sshd support querying remote revocation lists?

It doesn't AFAIK but it's not too hard to write a cronjob that `curl`s a secure endpoint occasionally (or `scp` or whatever).


That would be a Fail Open type solution which isn't exactly great where security is concerned.


Why would you in this analogy? The entire point of the distinction is you can't just temporarily add the mechanic's key to the driver's door, ignition, and bonnet.


there are facier cars that have a valet key that only open the driver door and start the engine..

I guess a similar solution could be implemented for a mechanical shop key..


Yeah, I use different keys for different purposes/projects but not for each and every server. Sounds inconvenient.

Plus how would only one of my ssh keys leak when I store them all in the same place? If I was worried about my keys leaking, I would rather do the YubiKey voodoo and keep my keys there.

But, again, once someone (persistently) compromises my main laptop, it's game over anyway.


> Did you know that ssh sends all your public keys to any server it tries to authenticate to?

Never really thought about it. Feels kind of "leaky", even if necessary.


Not a big deal at all considering that they're public keys, there's no security concern there.

Feels weird, but it's like going into a building that requires a badge and showing proof that you actually own several keys, then the building guard telling you he needs X key to enter since that's the one they know, and it's authorized.

All your public keys in your GitHub account are accessible through a link, just <github.com/<username>.keys>


> Not a big deal at all considering that they're public keys, there's no security concern there.

I might be pushing the analogy too far, but: all the URLs I have visited are public and are not identifying me personally, yet uploading them all together to a third party feels like a breach of privacy.


There is a privacy concern. And that might end up being a security concern depending on the threat model (e.g. social engineering attacks).


So, you'd ask someone to SSH into a server, and you'll get some of his public keys (I think the default limit is 5 keys), what would you do with that? You can also just go to their GitHub profile and fetch the keys or ask them to send you their public keys, they're meant to be public after all.

Here's one of my public keys, for free: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICIc3nOnSnsiAdkjAdH5WR9enQiPYWq1zAVsTDt60e91


As well as .keys, there's .gpg for GPG keys, and .png for the profile picture.


Host * IdentitiesOnly yes

To prevent


It's definitely a convenience over security feature. My personal feeling is that the convenience is worth it in this case, and it's a sensible default.



Really like the approach here. In the past, I've typically managed my SSH keys by naming them as id_<purpose> like so, then tinkering with the ssh_config to make use of them.

Never thought of using the token approach, though, definitely makes things simpler to work with.


The config file already does this, this is just a shortcut with %h and the file system structure.

You literally start an entry with "Host <hostname>" follwed by "User" and "IdentifyFile". There's even a bash autocomplete rule for it so you can tab through your servers "ssh <tab>". It won't send all the keys to the server if you organize this way (which doesn't really matter anyway, since they are PUBLIC KEYS).

It resolves to a preference: using a file or the OS filesystem to organize your keys.


> It won't send all the keys to the server if you organize this way (which doesn't really matter anyway, since they are PUBLIC KEYS).

Having a public key doesn't teach an observer your private key, and so they can't impersonate you, but it does allow the observer to distinguish you from others. If you would like to prevent observers from correlating identity this way (most famously if you use a public key for GitHub and also other things) you will want to explicitly forbid your SSH client from offering to prove identities other than the one you know will be used.

The setting in OpenSSH (which you can enable for individual Hosts) is IdentitiesOnly yes

Your proposed configuration will choose to try the file named, but it will not tell the remote server that you don't have any other identities, for that you need IdentitiesOnly. The default is no, although I guess it's possible you have overridden that to "Yes" previously and then forgotten.


> to distinguish you from others

Yeah, that's a good point. The less information that is leaked, the better.

> have overridden that to "Yes" previously and then forgotten.

Another good point. I have this at the start of my macOS config:

    Host *
        IdentitiesOnly Yes
        UseKeychain Yes
Edit: Uh oh, I think I misunderstood something! I'm still seeing nonexisting identify files being tried:

    % ssh -v whoami.filippo.io
    OpenSSH_8.6p1, LibreSSL 3.3.5
    debug1: Reading configuration data /Users/x/.ssh/config
    debug1: /Users/x/.ssh/config line 4: Applying options for whoami.filippo.io
    debug1: /Users/x/.ssh/config line 21: Applying options for \*
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/\* matched no files
    debug1: /etc/ssh/ssh_config line 54: Applying options for \*
    debug1: /etc/ssh/ssh_config line 58: Applying options for \*
    debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
    debug1: Connecting to whoami.filippo.io port 22.
    debug1: Connection established.
    debug1: identity file /Users/x/.ssh/id_rsa type -1
    debug1: identity file /Users/x/.ssh/id_rsa-cert type -1
    debug1: identity file /Users/x/.ssh/id_dsa type -1
    debug1: identity file /Users/x/.ssh/id_dsa-cert type -1
    debug1: identity file /Users/x/.ssh/id_ecdsa type -1
    debug1: identity file /Users/x/.ssh/id_ecdsa-cert type -1
    debug1: identity file /Users/x/.ssh/id_ecdsa_sk type -1
    debug1: identity file /Users/x/.ssh/id_ecdsa_sk-cert type -1
    debug1: identity file /Users/x/.ssh/id_ed25519 type -1
    :
Even though those don't exist, it is still trying them.


That’s because you have your SSH client config having to try them all.


But I have IdentityOnly: Yes for all hosts. Shouldn't that prevent it?


It appears that it's not so much attempting alternative identities as it's trying (potentially nonexistent) keys for different ciphers.


> You literally start an entry with "Host <hostname>" follwed by "User" and "IdentifyFile".

But that's so much pain.

> The config file already does this, this is just a shortcut with %h and the file system structure.

No.

    % ssh -F /dev/null -v whoami.filippo.io
    debug1: Trying private key: /home/moviuro/.ssh/id_rsa
    debug1: Trying private key: /home/moviuro/.ssh/id_ecdsa
    debug1: Trying private key: /home/moviuro/.ssh/id_ecdsa_sk
    debug1: Trying private key: /home/moviuro/.ssh/id_ed25519
    debug1: Trying private key: /home/moviuro/.ssh/id_ed25519_sk
    debug1: Trying private key: /home/moviuro/.ssh/id_xmss
    debug1: Trying private key: /home/moviuro/.ssh/id_dsa


Did you have an entry for "Host whoami.fillippo.io" in your config file?


My ~/.ssh/config has one line:

ForwardAgent yes

My public keys are in my dotfiles repo and my private keys were in hardware security keys/cards since 2016, unextractable.

I don't see any reason to do what this post is suggesting and in my opinion a post about SSH keys without advising to use security keys (which you also can use for 2FA) is a bit retrograde.


GitHub advises against this[0]:

> You may be tempted to use a wildcard like `Host *` to just apply [ForwardAgent yes] to all SSH connections. That's not really a good idea, as you'd be sharing your local SSH keys with every server you SSH into. They won't have direct access to the keys, but they will be able to use them as you while the connection is established.

[0] https://docs.github.com/en/developers/overview/using-ssh-age...


Just tried this

    $ ssh whoami.filippo.io
and it prints this

           You have SSH agent forwarding turned (universally?) on.
         That is a VERY BAD idea. For example, right now this server
          has access to your agent and can use your keys however it
                    likes as long as you are connected.

               ANY SERVER YOU LOG IN TO AND ANYONE WITH ROOT ON
                   THOSE SERVERS CAN LOGIN AS YOU ANYWHERE.
but I very much doubt that, because I didn't authorize my security key to log into this server, never mind any other afterwards.


Probably filippo should update this software to notice if the only identities presented were from FIDO authenticators and, if so, modify this message to explain the reduced risk

Note that although it's likely yours always requires a presence check (e.g. touch sensor), OpenSSH does not by default tell FIDO authenticators that it insists on UP (User Present) and so they are entitled (but few do since WebAuthn always asks for UP) to allow the signature to proceed immediately without a presence check so long as they don't set the UP bitflag in their signed response.

I think Fillipo's server can tell the client "I want UP" to check this, but I'm not sure and if few people do that I bet somebody already made a client which gets it wrong.

[[ Because the flags are signed, even though UP is a single bit flag you can't forge it ]]


For something like that to work an attacker would have to add into ~/.ssh/config (on each intermediate server) lines mentioned in another recent post

    ControlMaster        auto
    ControlPath          ~/.ssh/github.sock
    ControlPersist       999999999s
    ServerAliveInterval  0
without me noticing to sort of cache presence check. In which case they have access to my account already and presence or absence of 'ForwardAgent yes' makes no difference since they can add it if they want.

A paranoid answer to that is to use something like /usr/bin/ssh -F /dev/null -A user@fqdn.example.org everytime without an alias, of course. And only plugging in your security key when you need it.


Your description is very confusing, either I don't understand your explanation or you misunderstood.

ForwardAgent is a decision for your client, so an attacker's change to some intermediate can't cause your client to set ForwardAgent. Lots of modern SSH users do not have ForwardAgent, at all, it's just not necessary for them, so an intermediate server doesn't have the opportunity to do anything with it.

I also don't see how this is relevant anyway. My point was that although you will probably be asked to touch your FIDO token to authenticate to SSH servers, that's actually not technically the default, cheap tokens figured since WebAuthn is the majority use of FIDO, and since they're allowed to volunteer UP which WebAuthn wants, they can just ignore the UP flag on the request side and always do user presence testing. But the FIDO design does not require this, and so we can't know whether some/ all/ most tokens in say ten years time have this behaviour.


What I described there is a current, practical way to bypass subsequent presence checks in my particular security key (Google Titan NFC) which I tried out too.

What you're talking about is that some hardware implementations might be lax about user presence checks which is true, but doesn't affect any particular user (e.g. me) if they know that their key doesn't allow logging in without it.

> so we can't know whether some/ all/ most tokens in say ten years time have this behaviour.

I've used some U2F applets like this one https://github.com/tsenger/CCU2F which bypass presence checks, so yes I'm aware it's not a requirement, but it basically has to be a vulnerability on the hardware/applet level. So I guess don't use NONAME hardware is your advice, is that it?


No, it's not about a "vulnerability".

The FIDO Authenticator gets a request saying here's some parameters to sign, and most of it is much too high level for some cheap electronics to grasp (some of it is already just a SHA256 anyway, most of the rest can be just treated as bits with no particular meaning) but a handful of bitflags mean something to the authenticator. The authenticator must understand them, comply if able, and, if able and complying, set appropriate flags in the signed message.

The two we most care about today for end users are UP and UV. UP "User Present" means I checked there's a user present, e.g. I flashed an LED and somebody tapped the button. UV "User Verified" means I checked my owner is present, e.g. I have a fingerprint reader and the fingerprint matched.

Remember the main purpose people buy these things for is WebAuthn (the successor to U2F on web sites). For WebAuthn setting UP is always mandatory, your browser will, on every single request, set the UP flag, it always wants UP. It might set UV in some cases, but there aren't non-test public sites using this feature although it does exist in WebAuthn, it's obviously intended for the case where WebAuthn is both factors, e.g. something you have (the FIDO authenticator) and something you are (fingerprint to activate it)

Now, OpenSSH uses the same FIDO authenticators, but while Google is happy you are using the Google Titan NFC for this that's not why they made it. OpenSSH chooses not to set UP by default. A remote server (I think) can tell the SSH client hey, I need UP, get me UP or else you can't log in. But without that UP is not set in the signature request.

Most cheap authenticators today just ignore the UP flag in the request because it's easier to just assume it's always there since WebAuthn will always set UP. So even though OpenSSH actually says "No worries, I don't care if the user is present" these FIDO authenticators happen to require UP anyway, typically in the form of a touch sensor or hardware button. Since they're requiring UP I think they set the bitflag accordingly, if it was absent when required in WebAuthn you'd notice because nothing works

[The WebAuthn spec. calls out verifying this flag is present as one of the steps to validate a signed response]

You can definitely imagine a vendor focused on SSH ease-of-use would offer FIDO authenticators that care whether UP was requested and if it's not requested they don't need the extra press. For the vast majority of SSH users these products are more convenient, and if there's some case where they need UP the SSH protocol already can request that when you ask for it, so you should already be doing that if you want it, not relying on your device happening to do presence detection anyway.

I do not have a crystal ball. So that's why I can't tell if the present situation (they tend to do UP anyway because it was easier) is also what the future looks like.


I don't know how do you imagine a typical 'SSH user' looks like, but no company is out there racing to disrupt that particular field. There won't be a security key provider that cares enough to follow WebAuthn spec, but then decides to be lax about SSH usage for user convenience.

EDIT: looked into it a bit more and it appears that truth is somewhere in the middle. I didn't test it, but it's possible that as long as every chain link is fine with it you can opt-out out of user presence check.

https://wiki.archlinux.org/title/SSH_keys#FIDO/U2F

https://man.archlinux.org/man/sshd.8#no-touch-required

You need to have a SSH server that agrees to it, SSH key that is specifically made for this use case and maybe a security key that allows it to proceed. Probably worth testing if I find the time for it.


I was about to react to the version before the edit, but fortunately the edit was there when I hit reply, so, yes, exactly the thing you've found out.


> Probably filippo should update this software to notice if the only identities presented were from FIDO authenticators and, if so, modify this message to explain the reduced risk

The you can store non-FIDO keys on a hardware PGP device, use them for SSH authentication via GPG, and configure GPG to always require the same kind of UP checks for access to that subkey on the PGP smartcard. This gives you those same protections but doesn't show up to the server as any special key type.

As with the FIDO-based keytypes, you can (mis)configure this so it doesn't require UP checks at all.


> but I very much doubt that, because I didn't authorize my security key to log into this server, never mind any other afterwards.

What? This server pulls your public key from github and allows your client to connect to it. If you have forwarding turned on this server can INDEED connect to every server you are connected to.


No it cannot, not until I touch my security key. I'm not a novice in these matters and I just tested it again.

To connect to one server I have to touch my security key and then to connect to another one from it immediately afterwards I need to touch my security key again. Otherwise nothing happens and it just times out.

Even when using something like ssh -J user@server1 user@server2 I need to touch it two separate times.


Ah "security key" as in hardware token. Sorry thought you were referencing to your key file and not your usb-key. Misunderstanding that's all :)


Can you point to any existing write ups that are more in line with your methodology, for those of us that want to dig in and give it a try?


If you have Yubikey, then this is fine https://xeiaso.net/blog/yubikey-ssh-key-storage. I have Google Titan NFC which is basically Feitian K9 NFC, so it doesn't support -O resident.

Back in 2016 I was using https://github.com/philipWendland/IsoApplet which uses https://en.wikipedia.org/wiki/PKCS_11 standard, but modern way is much smoother as long as your OpenSSH server is up to date.


FWIW, the method in that xeiaso link (using resident keys on a security key) can be done with the cheaper "Security Keys" from Yubico, and doesn't need the more expensive Yubikey. -- The cheaper security key can be used as a FIDO key as 2FA for many websites; or even for stuff like Linux login.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: