
Stealing unencrypted SSH-agent keys from memory - lelf
https://www.netspi.com/blog/entryid/235/stealing-unencrypted-ssh-agent-keys-from-memory
======
yock
I've upvoted this because I want to see what really smart people have to say
about it. That said, I've always been of mind that if someone gains
unauthorized root access to my box, then nothing on it can ever be trusted
again. This includes things in memory. In other words, this feels obvious to
me, once pointed out.

Is there something novel about this that I'm missing?

~~~
tlrobinson
I agree with you, but in case you missed the author's rationale:

 _" However, this causes the attacker to have to wait for the target to type
in their passphrase. This might be hours, days, or weeks, depending on how
often the target logs out. This is why obtaining the SSH key from memory is
vital to pivoting to other machines in a speedy fashion."_

I'm not sure how you would solve this problem while providing the
functionality ssh-agent provides, aside from perhaps a HSM or something.

~~~
agwa
Anyone keeping SSH keys in an ssh-agent process for weeks on end is doing it
wrong. You can use the -t option to ssh-agent to specify a maximum lifetime
for identities added to the agent. I would set it to less than a day.

~~~
zobzu
if you unlock it as soon as it expires im not sure what that gives you. all
the attacker needs anyway in that scheme is 24h, its not a big difference.

------
nullc
I wish SSH had a narrow kind of delegation capability... E.g. if instead of
ssh-agent having access to the key, it used the key to sign a tuple like
{time-limit, [host-ip], ephemeral_pubkey}, then forgot the private key, and
used the ephemeral key to actually authenticate to hosts— but they'd only
accept it from the listed host ip set and during the specified time limit.

Obviously if the attacker has access long enough they'll capture the
passphrase and private key eventually, but it would be nice to narrow the
window.

Right now the best you can probably do is use a gpg smartcard with the ssh-
agent/gpg-agent shim stuff, (random tutorial here:
[https://github.com/herlo/ssh-gpg-smartcard-
config/blob/maste...](https://github.com/herlo/ssh-gpg-smartcard-
config/blob/master/YubiKey_NEO.rst))

~~~
cnvogel
Your scheme could probably be implemented using certificate based SSH: Your
users log onto a CA which gives out user-certificates with a specified
lifetime, they copy that to their desktop-machine(s), or maybe add it to a
ssh-agent like intermediate key-storage and can do their days work.

Tomorrow, they'll have to get new certs (automatically, after authenticating
to the CA using appropriate means).

Here's a tutorial showing cert-based ssh auth:
[http://neocri.me/documentation/using-ssh-certificate-
authent...](http://neocri.me/documentation/using-ssh-certificate-
authentication/)

Obviously with secure key storage (smart-card) prohibiting accidental loss of
the private key, and a working cert-revocation infrastructure it's not
necessary in the first place.

~~~
icebraining
That's essentially Mozilla Persona, but for SSH instead of the web. The people
at Mozilla are in fact already discussing such possibilities:
[https://hacks.mozilla.org/2013/04/mozilla-persona-for-the-
no...](https://hacks.mozilla.org/2013/04/mozilla-persona-for-the-non-web/)

------
e40
So, as someone needing to run unattended scripts across a wide variety of
hosts, what's the best practice for managing SSH keys? I've been using ssh-
agent. Is there a better way?

~~~
agwa
Make sure that the SSH keys are tightly restricted in the authorized_keys file
so they can only run the commands that need to be run unattended. If possible,
also restrict them by source IP address. Here's an example authorized_keys
file that I have in production for doing unattended backups:

    
    
      command="/usr/local/backups/backup_server /etc/snapshot_backup_list",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,from="10.70.0.0/16" ssh-rsa AAAAB...
    

I wouldn't worry unduly about protecting the keys themselves - since they need
to be accessible for unattended operation, there's not much you can do to
prevent them from being accessible to an intruder.

~~~
e40
This is hard, since there is such a large number of them. It would be very
painful to maintain this list. For a while I had the keys expire in ssh-agent,
but that didn't work because you can't extend the expiration before you run a
set of unattended commands. That is, if it's 11.5 hours into a 12 hour
expiration, and I need to run commands that will span the expiration, then I'm
screwed, since 1) I can't query when the keys will expire, and 2) I can't
extend it another 12 hours (from now, not from the expiration).

I feel like ssh-agent is essential to my life but it's badly designed and I
have this nagging feeling it's insecure (and articles like this feed this
fear).

~~~
agwa
That sounds like a bit of a quandary. One thing I'm not clear on - if these
run unattended, how do you enter the passphrase?

ssh-agent isn't insecure, it's just not magic. If an attacker gets root access
to a box, they can examine the system's memory, and the memory of ssh-agent
necessarily contains the unencrypted private keys.

Edit: if you're really concerned about private keys being exfiltrated, you can
always use a smartcard (the OpenPGP card[1] in a Gemalto USB Shell Token[2]
works well with SSH). But if the smartcard is online all the time, then an
intruder can always simply _use_ the smartcard to SSH wherever they want, even
if they can't actually get the private key itself.

[1]
[http://shop.kernelconcepts.de/product_info.php?products_id=4...](http://shop.kernelconcepts.de/product_info.php?products_id=42&osCsid=bdf5854d9214419637adcf0c5215e6fe)

[2]
[http://shop.kernelconcepts.de/product_info.php?products_id=1...](http://shop.kernelconcepts.de/product_info.php?products_id=119&osCsid=bdf5854d9214419637adcf0c5215e6fe)

~~~
cortesoft
I think the point is that they enter the passphrase once to load the keys into
ssh-agent, and then the unattended process can use them forever.

~~~
agwa
If that's the case, then I would start a dedicated ssh-agent process just for
the run, add the identities (without expiration), and later kill the agent
when the unattended run is over.

~~~
__david__
That's fine for that kind of scenario, but often "unattended run" means
something spawned by cron, like a daily backup, or a CI system that spawns
build jobs on slave machines. There's not a lot of options there.

------
ipsin
The technique of using gdb to attach to ssh is pretty old.

Here's an example from 2005, which was presented (iirc) at Defcon as well:
[http://www.blackhat.com/presentations/bh-usa-05/bh-
us-05-boi...](http://www.blackhat.com/presentations/bh-usa-05/bh-
us-05-boileau.pdf)

In that case, SSH-Jack would just piggyback on existing (user-level) ssh
connections, which is also pretty serious, though that's not as exciting as
stealing keys.

~~~
void-star
Yep. The general technique of scraping memory for private keys and
certificates a pretty well-known and established. Particularly in forensics
and malware circles.

I don't point it out to engage in the "who was first" thing. But to point out
that this is very much an applied attack in the real world. Real attackers
(includes "forensics analysts", incase you don't consider them "attackers"
too) have been using this technique in malware as well as
countermeasures/investigations for quite a while, now.

See:
[http://www.trapkit.de/research/sslkeyfinder/](http://www.trapkit.de/research/sslkeyfinder/)

[https://github.com/emonti/yara-
ruby/blob/master/samples/sslk...](https://github.com/emonti/yara-
ruby/blob/master/samples/sslkeyfinder)

[http://volatility-labs.blogspot.com/2013/05/movp-
ii-21-rsa-p...](http://volatility-labs.blogspot.com/2013/05/movp-ii-21-rsa-
private-keys-and.html)

EDIT: actually a much earlier discussion is from '98 by none other than Shamir

[https://www.cs.jhu.edu/~astubble/600.412/s-c-
papers/keys2.pd...](https://www.cs.jhu.edu/~astubble/600.412/s-c-
papers/keys2.pdf) [PDF]

------
Canada
This is exactly why SSH has agent forwarding.

Do not copy your private key to other systems.

ssh -A is your friend.

~~~
m-app
Well, it prevents your private key from being compromised but an attacker with
root on the intermediate box can still authenticate to other servers using
your SSH agent socket.

If you really want to do secure SSH hopping, use the ProxyCommand directive
with `netcat` in your SSH config.[1]

[1]:
[http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Ju...](http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#ProxyCommand_with_Netcat)

~~~
Canada
Nice. Thanks for sharing

------
ambrice
So, is there a big advantage to running ssh-agent vs unencrypted 0400 keys in
your ~/.ssh directory?

~~~
agwa
Yes. ssh-agent prevents non-root users, _including the user running ssh-agent_
[1] from being able to extract the private key. That's much more secure than
storing unencrypted keys in ~/.ssh.

[1] ssh-agent uses some tricks, like making itself setgid, and/or using
prctl(PR_SET_DUMPABLE, 0) on Linux to make its memory undumpable.

~~~
0x0
And on OSX, it integrates with the keychain as well!

~~~
void-star
You're better off not storing your passphrase in the keychain and just typing
it in that one first time, though. But it's still marginally better than an
unencrypted private key for automation if you need a 100% unattended setup.

~~~
lloeki
"In the keychain" doesn't necessarily mean in the _login_ keychain. You can
create as many user keychains as you want, with different passwords. Also, add
the keychain app to your menubar (via preferences) to get a quick way to
manually lock one or all keychains (and alternatively, the whole computer).

