
Sshmuxd: SSH proxy to replace jump hosts - Spiritus
https://github.com/joushou/sshmuxd
======
jamiesonbecker
Jumpboxes aren't that bad to automate, actually.

We already help with automating jumpbox creation. (docs:
[https://userify.com/docs/tips/jumpbox/](https://userify.com/docs/tips/jumpbox/))
and we're building even more jumpbox automation now. Don't allow root for any
jumpbox accounts, but of course root escalation exploits abound. (I just found
another in an AWS agent yesterday.)

Of course, as another commenter mentioned, if your jumpbox is compromised,
than the jumpbox could serve as a gateway to your network. It's a tradeoff
between exposing all of your servers to inbound SSH or only one.

There's another way, which is pure TCP forwarding on a different port for each
server (ie 21321 -> inner server 22), but whether this actually reduces the
attack surface is debatable, since the totality of open ports remains the same
across the entire network.

My personal feeling is that using a jumpbox and locking it down (preferably to
your company's IP ranges, etc) is the best way to go. You can also add MFA to
the jumpbox entry point itself. We're going to help with automating all of
that in the near future, too.

(disclaimer: CTO @Userify)

~~~
joushou
Hi CTO for Userify person!

One thing is setup of jump hosts (basic version just being a box with sshd
enabled + users with authorized_keys, which is beyond simple), but it's the
limitation of hosts. I have 1 big network where a set of users are only
allowed to jump to 1 or more unique host each, with myself and a few others
being able to jump to every single one. I believe that your solution requires
things to be on different subnets to isolate them, with separate jump hosts
for each subnet, correct? Here, I only need one sshmux for everyone.

Another "fun" feature is that sshmux can also throw unknown users to a
separate host, such as ssh-chat, for support or hanging out.

Actually, in case of -oProxyCommand="ssh -W %h:%p jump_host", you're still
fully secure in the case that the jump host (sorry for not calling it a
jumpbox) is compromised. This is true both for normal SSH servers and for
sshmux. They can mess with the original raw ssh connection that requests the
forward, but this will just screw the connection up. They can connect it to a
wrong host, but that will provide incorrect host keys. The agent isn't
forwarded, so they don't get to sign anything with the private key. Messing
with the new connection that gets established is subject to SSH's normal MITM
resistance. This means that even root on the jump host would require real SSH
protocol vulnerabilities to attack the connection.

Good luck with Userify!

~~~
jamiesonbecker
That's cool. Another way is to localtunnel to TCP forward to endhost in your
config (ie two SSH connections, the first to the jump host, the second through
the jump host, without agent forwarding), but that gets tedious fast without
scripting support, especially if you want to use different portnums to avoid
conflicts. ControlMaster would be a good choice to multiplex multiple
connections to the same box.

> I believe that your solution requires things to be on different subnets to
> isolate them, with separate jump hosts for each subnet, correct? Here, I
> only need one sshmux for everyone.

Right, we're automating the .ssh/authorized_keys (and sudoers) right now
([https://github.com/userify/shim/blob/master/shim.py](https://github.com/userify/shim/blob/master/shim.py)),
so you can set it up however you like, as long as those boxes can get to
either Userify for key updates, or your on-premises Userify server(s). Only
one jumpbox is needed across any network, depending on how you prefer to
lockdown your network. (The HOWTO refers to EC2, but it can be built anywhere,
exactly the same way.)

sshmux looks like an interesting project, and love the simplicity. Wonder if
Userify might automate the ProxyCommand setup! Seems really useful.

> Good luck with Userify!

Thanks! We have 350 companies on the platform right now and growing fast, and
Userify Enterprise (on-premise) is in beta at several large enterprises as
well.

BTW, joushou, we're looking for brilliant SSH and golang hackers.. ping me if
you get bored :)

~~~
joushou
> sshmux looks like an interesting project, and love the simplicity. Wonder if
> Userify might automate the ProxyCommand setup! Seems really useful.

Yeah, ProxyCommand is a bit cumbersome for many setups, which is why the agent
forwarding mode has the interactive selection screen...

If you feel like poking at sshmux or think there's anything that would be
helpful if added, don't hesitate to add it to the issue tracker. Real-life
use-cases get higher priority.

> Thanks! We have 350 companies on the platform right now and growing fast,
> and Userify Enterprise (on-premise) is in beta at several large enterprises
> as well.

That's great! I guess getting Userify Enterprise out of beta is a bit thing on
the roadmap.

> BTW, joushou, we're looking for brilliant SSH and golang hackers.. ping me
> if you get bored :)

It sounds very interesting (I like Go and SSH, what can possibly go wrong?!),
and I am indeed occasionally bored, but I'm ~8000km away... Commuting from
Sweden to Denmark is bad enough! ;)

------
daurnimator
> sshmux, and by extension, sshmuxd, can only forward normal sessions (ssh'ing
> directly to sshmuxd without a ProxyCommand) if agent forwarding is enabled.

This is very dangerous.

With the average setup, anyone with root access on the middle box can borrow
the ssh key of any user connecting through it.

See [http://unixwiz.net/techtips/ssh-agent-
forwarding.html#sec](http://unixwiz.net/techtips/ssh-agent-
forwarding.html#sec) for more info.

~~~
hlieberman
Considering this seems intended for use in somewhat corporate situations, it
may or may not be that serious of a problem.

This is yet another situation where Kerberos got it right years and years ago,
but is almost impossible to actually /use/ even now after n years.

~~~
joushou
Exactly. And yes, Kerberos got it somewhat right, but as someone who have used
configured and used it, it is quite the exercise to set up. It took ages to
get working, and even then, it was slightly cumbersome to use.

~~~
cpach
_" It took ages to get working, and even then, it was slightly cumbersome to
use."_

I've wondered from time to time why Kerberos more or less faded into
obscurity. I guess we have the answer then.

------
Galanwe
I don't really understand what this is supposed to be useful for. Relying on a
random software to secure your ssh entry point, instead of a proper linux
configuration, that seems like a risky tradeoff.

~~~
joushou
I'm the author of the project.

The project is meant to ensure that you can allow multiple users access
through a jump-host style mechanism, while not permitting any other "abuse" of
the jump host. You can lock down SSH a lot, but not as much as sshmux does.
Security wise, the code is very, very simple and easy to follow, and even if
it went rogue, that's no different than the trust you put in your usual SSH
server. If this is a concern, do not use the agent forwarding mode, which
would render a rogue server a pointless and unfruitful prank.

If you use the ProxyCommand mode, the only thing an "evil" sshmux would be
able to do would be to break the connection. It won't be able to fake the
endpoint if it is already in your known_hosts, as it does not have the real
endpoints private host key. It is, therefore, secure in this mode.

------
Daviey
Increasingly I am using sshuttle[0] to solve this problem. When you have lots
of machines, ProxyCommand'ing always feels like such a burden.

[0]
[https://github.com/apenwarr/sshuttle](https://github.com/apenwarr/sshuttle)

~~~
joushou
When I use SSH as VPN, it's usually because I want less flexible applications
to use it, using socks5 configured as a system-level proxy. When doing more
work on our corporate network, I usually SSH through the jump to my desktop
and work from there, unless I specifically need to access another machine for
something.

As for sshuttle, it's a very convenient device, although the design appears
rather complicated. I'll look into it.

~~~
beagle3
The design is possibly the simplest for what it does (and it does that
exceedingly well).

sshuttle is:

\- a tcp multiplexer (multiple connections onto one stream)

\- a router (uses firewall rules to make normal connections go through the
multiplexer)

\- a few kludgy tunneling operations, to make the combination above seem like
a real VPN (discovery of remote subnets, tunneling of DNS requests, ....)

\- a default setup that runs the multiplexed stream through an ssh connection,
thereby giving all the security guarantees that ssh provides (integrity,
confidentiality, mitm resistance)

\- packaged in such a way that the remote side needs to have a minimal
python>2.6 install, and a user capable of making tcp connections. Nothing
more.

In my experience, it works way better than IPSEC tunnels and some commercial
VPNs that I've used. With two caveats that I'm ok with with: a) only TCP, and
b) all connections seem to come from your remote ssh server. No VPN solution
that I'm aware of makes so few demands; do you know of one that doesn't
require root at the remote server?

~~~
joushou
ssh -D (socks5 mode) doesn't require root, and can forward both udp and tcp
traffic, doing what it appears that the client requires. socks5 is just a
protocol that tells the server to connect to X on port Y using protocol Z, or
to bind to X on port Y using protocol Z. Root is only required if you want to
bind to privileged ports. A bonus with this is that it only requires a SSH
server in default configuration.

SOCKS5 most common usage is to just use it as a proxy for HTTP traffic or
similar, but it can make any connection. The trick lies in the "kludgy
tunnelling operations" and the "router" (firewall rules).It's this magic that
makes it better than the socks5 proxy solution, as it really does appear like
a VPN. I wonder how much effort it would be to get rid of some of the "magic",
though.

But I have to agree, that _ANYTHING_ is better than a real VPN in complexity.

~~~
beagle3
> ssh -D (socks5 mode) doesn't require root, and can forward both udp and tcp
> traffic

No, but it requires a SOCKS5 client, of which there aren't many among software
I use.

> The trick lies in the "kludgy tunnelling operations" and the "router"
> (firewall rules)

The other important part of the magic is the packaging. While ssh+socks
support is pretty common these days, it is not supported on e.g. dropbear
(which is popular on routers).

sshuttle copies whatever it needs to the other side; as a result, there's
never an issue of version compatibility (which socks use sidesteps by virtue
of using an old, cemented protocol). I have, in fact, needed to modify
sshuttle with new options and features for a project, and I was not bound in
any way by the existing protocol - whatever changes I made were always
supported by the other end of the connection because the connection starts by
copying them to the remote.

> I wonder how much effort it would be to get rid of some of the "magic",
> though.

sshuttle is very well written; It is easy to tear it apart and rebuild it in
other ways. But if you took the magic apart, you'd be left with a non-standard
UDP-deficient SOCKS-alike system. Why would you want that?

~~~
joushou
If you tied the local FW magic to a socks5 client, you'd be able to skip part
of the magic, assuming you don't desperately need mDNS. SOCKS5 does UDP too,
IIRC. I like tearing things apart, and I like making clean muggler solutions,
rather than relying on wizardry. :)

With that said, I might still end up using sshuttle as it is. It's a nice
project. I just crave for tinkering around. :D

------
erikb
Why not simply configure your ssh correctly?

~~~
guipsp
Jump hosts exist for a purpose.

~~~
erikb
Yes, and there is a way to configure your client to use one.

~~~
joushou
sshmux is a jump host that allows more user-control than you can with a ssh
server. Unless you use the interactive mode with agent forward, you still need
to configure your client to use one.

I'm not sure I get what you're on about, to be honest. :/

------
tokenizerrr
Does this support SFTP? What about for windows users that use FileZilla and
don't have an .ssh/config?

~~~
joushou
SFTP/SCP works if you use the ProxyCommand, or agent forwarding. Clients such
as WinSCP support agent forwarding, at least. Not sure about FileZilla. I'm
not a Windows user, so can't say. :/

~~~
tokenizerrr
I know FileZilla also supports agent forwarding, but I'm not entirely sure how
this would work in practise. If I'm understanding things right you can have a
single jumphost and give a user access to multiple targets. So when the user
connects to the jumphost using SFTP then what files/directories do they see?

Maybe I should just install and try this for myself...

~~~
joushou
Interactive selection for agent-forwarding only works for regular SSH, not
sftp, due to having no way to enter input.

If I get around to it, one could wrap sftp completely, so that the available
servers simply show up as root-level directories.

EDIT: Opened an issue about it.

~~~
tokenizerrr
> If I get around to it, one could wrap sftp completely, so that the available
> servers simply show up as root-level directories.

That wound be amazing! I'll see if I can check if ProxyCommand is doable with
FileZilla later as well.

------
thomashabets2
[https://blog.habets.se/2014/06/Another-way-to-protect-
your-S...](https://blog.habets.se/2014/06/Another-way-to-protect-your-SSH-
keys)

Seems like the same thing. Even written in go, too.

~~~
joushou
I see why you might think that, but no. SSHProxy tries to move the private key
from your computer to an external thing. SSHProxy doesn't have anything to do
with sshmux.

SSHProxy is a two part thing:

1\. A server that, when connected to, will ssh to the requested host, but
authenticate with a local private key.

2\. A client that will, when connected to the server, work like netcat for
ProxyCommand purposes.

I'm skipping some info about additional SSH sessions being involved, as they
don't provide any additional security.

The result is essentially ProxyCommand="ssh -W %h:%p jump_host", but instead
of you having the private key, the jump_host has it. You then use a different
private key to authenticate to the jump host.

I get the idea, and I by no means say he shouldn't have implemented it, but
this provides no additional security what so ever. It just moves your private
key from one machine to another. I hope he had fun implementing it, though! If
you don't implement what might be a bad idea, you'll never get around to the
good ones. Plus, bad ideas can be fun, even if they remain bad when done.

~~~
thomashabets2
Ah yes, on closer inspection on a non-phone they are quite different.

You are just wrong on the no additional security though. In addition to
enabling auditing, it also allows a username/password to be turned into an ssh
key, so that you can know that if the rpi wasn't turned on, then nobody could
log in to the target machine. It's similar to a TPM in this regard (as blog
says), or can be a jumpgate protecting the systems behind it from compromised
or malicious users.

I'm sorry you don't get it, but I'm not hurt by someone not understanding
something when they criticize it. If you don't see the point of not having
direct access to your key, then there's nothing I can do.

~~~
joushou
I have to disagree with you.

1\. The private key is not secure by default by putting it on a different
host. 2\. This still allows an evil user getting access to his original
machine to SSH into any host requiring his private key.

1 should be pretty obvious. If my intention is to get a hold of the actual
private key file, he just moved the target. As for 2: We're talking about
avoiding compromising a private key file stored on the client, so in order for
that to be an issue in the first place, we're giving an evil user Y reading
abilities on the local client. The normal user is called user X. The evil
thing to do is to log into the other server on poor user X's behalf. User X
used SSHProxy, and moved his private key to a raspberry pi. So, what does evil
user Y do on user X's computer?

He just logs in through the SSHProxy, getting access to the remote service.
Having the raspberry pi disconnect at times doesn't really matter, as that
just means patience (or starting a job that keeps trying). Once logged in,
evil user Y could add another key to the authorized_keys file if he wanted to.

Having a service automatically sign things with private key A stored in remote
location B, as long as you provide private key B stored in "easily accessible"
location A (Which we still assume due to thinking the private key was
insecure) is no more secure than storing private key A in "easily accessible"
location A. The only argument becomes the ability to occasionally take the pi
offline, which doesn't seem to be what he touts as a reason for the project.
The issue with this is essentially a reversed modified version of the ssh-
agent socket issue, which is blindly signing things as well.

If you truly want to protect your private key: * Encrypt a USB dongle * Put a
password protected private key on it * Only plug said USB dongle in when you
need it, store separately from your laptop (on your person) * Change your
password occasionally

Likewise, having a TPM be your signer doesn't change anything if you have it
sign blindly. PCR's allow you to at least verify system state first, but using
the TPM in this case only protects you against people stealing your filesystem
without your physical machine. With access to the machine, the TPM will keep
on signing things blindly. User interaction beats TPM in all areas. One of the
cute features TPM was supposed to bring, that actually was going to matter (if
you ask the manufacturers, that is) was Remote Attestation. Unfortunately, it
wasn't any good and was never used for anything.

Also, for future reference, while it's completely okay to suggest that people
are wrong or misunderstanding, there are nicer ways to say it than "You are
just wrong", and "not understanding something when they criticise it". I have
no idea about your qualifications, and you have no idea about mine. We're all
humans, so be polite about it. No one likes being insulted.

~~~
mst
> Also, for future reference, while it's completely okay to suggest that
> people are wrong or misunderstanding, there are nicer ways to say it than
> "You are just wrong", and "not understanding something when they criticise
> it".

It's useful information to me that you value pleasant phrasing over criticism
that might improve your security.

Thanks for making this clear up front.

~~~
joushou
It's also good to know that you completely ignored security advice from
security professionals, making this a waste of time. You could say that up
front as well: "I'm going to write a comment insulting you, but don't you DARE
try to disprove me with logic!"

------
j_s
I've used a similar setup to wrap RDP to allow employees to securely remotely
access their office desktop. There is a market for someone willing to make
this a turn-key replacement for Terminal Services.

------
VLM
Probably 15 years ago I did something similar at an overall system level with
a program called pdmenu which provided a jump host menu for semi-technical
users. The technical details of course are entirely different, but it did the
same general system idea of presenting individual users with custom tailored
prompts to log into various systems (and a few other tasks, and logging, etc).
pdmenu had (has?) a great menu CLI for end users.

------
wila
Looked at the code, it is nice and short.

However I'm not that familiar with go so might be overlooking something. Is
there any logging included on who connected at what time from what IP?

~~~
joushou
Not at the current time. Will implement, just haven't had the time. I mainly
code on this during my daily commute between Denmark and Sweden. :)

~~~
tinco
You live in Copenhagen and work in Malmo or something? Is that common?

~~~
joushou
Landskrona, actually. Malmö would have been nicer, but the waiting list for an
apartment is 2-3 years.

It's actually quite common, especially in Malmö (Malmö<->Copenhagen takes
about half an hour, Landskrona<->Copenhagen 1:20), for a lot of reasons. The
first being that living expenses, including rent and household items, are
cheaper in Sweden. Local wages are always adjusted after the local living
expenses, so living in Sweden while working in Denmark is economically
favorable, especially seeing that you get a tax deduction bonus based on
distance from home to work through the shortest feasible route. Cars are also
orders of magnitude cheaper in Sweden due to different taxing. Not everything
is cheaper or better there, though. Diesel and alcohol is more expensive, and
my selection of rye bread is much narrower up there!

For me, it's due to me wanting to start a life with my wife, and a lot private
things going on that made Sweden the obvious choice.

------
zippie
This is nice but the agent forwarding is a legitimate concern especially
because a compromised host can take off with the private keys which are used
while beginning the session. Most private keys are used for more than
connecting to a single host.

In our setup we use jailkit allowing only ssh passthrough [1], we have added
LDAP support to it (may release the patch later).

[1]
[http://olivier.sessink.nl/jailkit/howtos_ssh_only.html](http://olivier.sessink.nl/jailkit/howtos_ssh_only.html)

~~~
joushou
What authentication do you have from the jump host to the target? To me, it
looks like it has been reduced to either none or keyboard-interactive
(password) login, which is considered bad practice. I could very easily
implement support for this in sshmux, providing the setup you seem to use, as
a way to avoid agent forwarding, but I just genuinely did not expect anyone to
use password authentication, apart from in default config scenarios, before a
public key has been installed.

Agent forwarding does not provide private keys, but only individual signing
requests. This means that while the user is connected, and evil remote can
request arbitrary signing, but only as long as the user is connected, and only
as long as the users ssh agent is willing to do so. Using ssh-add -c further
means that the user will have to accept each signing request. Also note that
ssh -W, which is immune to any of these concerns, is fully supported by
sshmux.

Correct me if I'm wrong, but "Jailkit" does not seem to stop an evil
legitimate user from poking around other hosts with forwarding and such (which
they can as long as ssh -W functions, within the limits of what a firewall
allows the jump host to do in general). sshmux allows you to lock users to
targets, rather than lock general capabilities of the entire jump host. This
is because, while I provide clients access to various hosts, I do not trust
them enough to have any unnecessary privileges.

------
zobzu
Its nice.. That said, one if the real issue is getting people to use -W ;)

~~~
dijit
I'm looking at the man page, I had no idea about `-W`. However, what does this
protect against that the standard:

    
    
        ProxyCommand ssh -q <jumpHost> nc %h 22
    

doesn't protect?

 __Edit: __just tried `-W` as an addition to the line but I get

    
    
       littlefish :: ~ » ssh pgsql3
       Bad stdio forwarding specification '<jumpHost>'
       ssh_exchange_identification: Connection closed by remote host

~~~
keeperofdakeys
Try formatting it like this:

    
    
        ProxyCommand ssh jumpHost -W %h:%p
    

-W is essentially the same as using nc, however it doesn't require executing a command on the jumphost. This project uses that fact to make the jumphost more secure (you can't execute arbitrary commands), and do destination white-listing for each user.

------
james_woods
Why not simply use a VPN?

~~~
joushou
VPN's can be quite a pain, and is considerably more work than raw SSH.
Depending on the VPN, the authentication strength is usually also quite a bit
lower than that of SSH with RSA keys. There's also additional overhead. I have
both VPN and jump hosts to get into our corporate network, and I most
certainly prefer the jump host for convenience and performance.

There is also the user-level access restrictions of sshmux which will be much
more difficult to replicate with VPN's. At least with pptpd, I believe it
would require putting users on individual subnets with firewall rules
restricting access to only their permitted hosts. Long story short, it
wouldn't be a feasible solution.

