Hacker News new | past | comments | ask | show | jobs | submit login
Why aren’t we using SSH for everything? (medium.com/shazow)
270 points by dkua on Jan 2, 2015 | hide | past | favorite | 103 comments

For one thing, SSH performance for large transfers is abysmal over high-latency links. SSH uses a small TCP window size because it was optimized for quick response, not bulk transfers. (After all, it was designed to be a remote shell protocol, not a generic file transfer protocol; scp and sftp were added after the fact.) And unfortunately you can't currently specify a different buffer size even if you wanted to.

More information can be found here: http://www.psc.edu/index.php/hpn-ssh

> scp and sftp were added after the fact

scp was there from the beginning, or thereabouts. It certainly was present the first time I played with it which was within a few months of the first release.

ssh was specifically designed as a replacement for BSD remote tools (rsh/rlogin/rcp) even falling back to those protocols if ssh wasn't available (although the FallBackToRsh option was removed from OpenSSH a decade or so ago I think). That's also why "slogin" is installed as a symlink to "ssh", to keep it similar to rsh/rlogin. It only made sense for there to be a "scp" to work like the existing "rcp".

You are right that sftp only came later.

Are there any plans to patch this into OpenSSH? It looks like they have a great solution for the problem you highlighted.

Those patches are old. My understanding is that the parts that are reasonable have been implemented and the parts that aren't (plaintext modes) have been ignored.

If you need to transfer large files around a lot use what the supercomputing centers use: GridFTP.

I'd never even heard of GridFtp before. It looks pretty good. I've taken to not installing ftp on servers at all and just using ssh, but this might make me think again.

A solution would be to have some kind of protocol revolving around ssh, that would open one ssh session, and use other socket connections to do other stuff, like ftp, scp, wget, etc...

Would be pretty hairy though. Maybe another solution would be to have some new socket standard that can achieve what both UDP and TCP do, at some kind of variable window size...

Something else worth mentioning; SSH has built-in support for single-sign-on, via SSH agent forwarding. As long as my public key is available anywhere (and that's exactly what it's designed for), then I can be authenticated by any system, anywhere. Thus, a problem which is so vexing in so many other scenarios, is very cleanly addressed by SSH.

Fundamentally, a password is a shared secret. So you send your password to a server, you are trusting that server not to lose or misplace it. In contrast, an SSH public key doesn't require nearly such careful management.

SSH agent forwarding is extremely risky. Anyone with appropriate permissions (legitimate or illegitimately gained) on a machine you have connected to can use your credentials to open a new connection to any machine you have access to.

This is a very valid issue. I have been using ssh-ident* for this reason. You can add keys to ~/.ssh/identities and it'll start up an agent for each identity when you use it. This means if the machine you connect to is indeed stealing your keys, at least they only have one key, not all the keys you're currently using.

There are some caveats; such as having to specify what domains/IPs should use which identity, and also having to alias ssh to get it to work nicely. So it's not perfect, but it does solve the problem you talked about.

* https://github.com/ccontavalli/ssh-ident

> on a machine you have connected to

Don't you have to be actively connected to the machine for this to work? i.e. the server I haven't connected to for a few months has no way of opening connections on my behalf at this point, right?

You are right.

But once you connect, a year later - in those 3 minutes before you disconnect, the attacker might have authenticated as yourself to 100 other machines -- and appended their own key to .ssh/authorized_keys on these machines, so that the compromise no longer needs you to be connected.

You are correct. That is what I meant, but I should have said "a machine you are actively connected to."

Agreed! On nomenclature - you're saying ssh agent forwarding, but the functionality you describe is actually simply public keys on remote machines. The only time you need to forward is when you're logging into another machine through a third machine -- and as other commenters have pointed out, you're extending your trust model to that third machine.

(blatant plug) Userify can distribute those public keys and reduce (but not eliminate) the cases where agent forwarding is required.

Of course, you need to trust the remote systems...

Isn't there a problem when you tunnel TCP over TCP with increasing window sizes (auto throttling mechanism meant to prevent packet fragmentation)?

Every time I've tried to keep a long-running ssh tunnel for printing / http, the connection degrades after a while. I'm sure there are some flags that can be set, but I thought this was the major show stopper for the "everything over shh" (since ssh uses TCP protocol)

It depends how you set up the forwarding.

Casually looking at strings /usr/sbin/sshd on my devserver reveals that there is a mode to create a tun device, which sets up a more traditional VPN-ish thing (basically, your computer sees a new interface and routes certain sections of IP space to that prefix. sshd sucks in these packets and forwards them as is). This results in two layers of congestion control, which (IIRC -- I'm not a TCP guy) Congestion Control Algorithms aren't set up to handle very well.

The more traditional mechanism for tunneling though (the one most often referenced in the posts and things I've seen) is a layer 7 forwarder. Basically it creates sockets to listen for connections on either ends on a specific port, recv's data as a traditional network user, and then turns around and sends the data over the multiplexed transport to the end server. It essentially strips away the congestion control on the first hop (because you're just going to localhost anyway) and there's no TCP/IP packets in TCP/IP packets thing going on anymore, just your application stream in their header transport thing. It's less flexible, but is good enough if you want to set up a simple proxy or something and doesn't have the same pitfalls.

If you need the flexibility, something like OpenVPN might be good (though I haven't used it myself).

And you might also want to look at stunnel as an ssl tunnel alternative that lets you have more control over what certs go where without archaic and dangerous /etc/ssh/*_config meddling

Simple port tunnelling (-L and -R command line arguments) do not suffer from TCP over TCP. And if you want VPN-style usage, look into sshuttle. It is one way only, and that's a good thing! (Most of the time you only need connections going one way and none going the other way). If you do want two way, either use two sshuttle connections (one each way) or OpenVPN.

The only thing about sshuttle that I've encountered that exposes it's non-VPNness, is that the connection truly originate from the remote system - e.g. if you connect through sshuttle to your peer, the connection go from the peer to the peer on That may or may not be a problem (e.g., logging is much less useful this way).

Try sshuttle. I've stopped using VPNs since I started.

I have to agree that VPN's are usually a bit too much; generally speaking, VPN's are a chainsaw when a scalpel is more appropriate!

Have you tried the autossh utility? That's what I use to keep up long-term tunnels.

The window size issue has to do with bandwidth, not session duration. See, e.g., http://www.psc.edu/index.php/hpn-ssh

Why aren’t we using SSH for everything?

Because "Use X for everything" is a terrible design decision? SSH uses flexible transport with some desirable features and may be underutilized in practice.

This question is starting to feel like people who want to staple every pie in the sky idea to the bitcoin blockchain because it too has a set of desirable properties.

You're right, we probably shouldn't use SSH to microwave our food.

But a lot of things where we use HTTP today, we could be using SSH if we had better library support. Some more ideas towards the end of the post.

Someone has to be running NetBSD on their microwave owen. I remember a toaster story.


> we could be using SSH if we had better library support

It's the firewall rules, not library availability.

Tunnel SSH over HTTP, best (worst) of both worlds :)

(I have tried this; it is both useful and terrible)

Or you could just run `sshd` on port 443.

But this would fail on many proxies, and any firewall that was aware of the protocol.

Indeed. And more generally, using some of the SSH concepts on the web such as public key authentication.

Neat trick, if you're so inclined to use such tricks:

    $ cat .ssh/authorized_keys
    command="tmux new-session -A -s base" ssh-rsa [...]
Automatically creates or joins a tmux session named base, and disconnects the SSH session when you disconnect from the tmux session.

So, yeah, why don't we use SSH for more?

It's multiplexing support uses static sized windows, which even on modern LANs means you usually only see 3-5 mb/s transfer rates where you should be able to - even with encryption overhead - achieve almost gigabit NIC speed.

Fortunately the HPN-SSH patches exist to solve this problem - but I really want to know why their's so much resistance to adding them upstream.

> which even on modern LANs means you usually only see 3-5 mb/s transfer rates where you should be able to

On a WAN, sure, but even a mid-2000s gigabit LAN could routinely hit 800+Mb with scp as long as you had tuned the underlying TCP stack (Linux was poor and OS X/*BSD worse in that era) and weren't using something slow like 3-DES.

> Fortunately the HPN-SSH patches exist to solve this problem - but I really want to know why their's so much resistance to adding them upstream.

Looking at the patches, I'd be surprised if the problem wasn't the fact that they change other things with security or reliability implications. Seeing something like “Dynamic Window and ability to use NONE encryption” suggests that it'd be better to break it up into some smaller generally-useful patches and a separate patch for people in controlled environments who need as much performance as possible.

It is broken up into separate patches as well (scroll down the homepage).

But the NONE encryption makes a lot of sense in a "use SSH more" perspective. When you need to move piles of totally non-private data, but want to use a secure authentication mechanism (and message authenticity system) for issuing shell commands...

Good answer.

If I had to guess, the reason would be threefold. One, it requires tuning on both ends of the connection to make the high speed transfers work well (and it looks like it will interfere with multiplexing as well, the reason the fixed window exists in the first place).

Two, the "none" cypher which it enables completely undermines the security of the secure shell. The cypher renegotiation after authentication would make it trivial to MITM the connection. You could probably disable the cypher after every transfer by restarting the sshd daemon, but that's being a bit optimistic.

Finally, there is no visible license on the patchfile, or on the homepage. The project's sourceforge page lists it as being BSD licensed, but there is no indication from the author that this is the case.


> means you usually only see 3-5 mb/s

I see over 50Mb/s (=~400Mb/s) on a gb network without any attempts at optimization, between a stupid Atom machine and a fast machine, both running Linux (which other responders claim is unoptimized). When did you last test?

Also, this might be the speed of the disks at one of the ends - I've never even bothered to check, because making it faster doesn't make a difference for me.

My experience with a 20-disk RAIDZ3 machine and my desktops has always been that SSH tops out at 3-5mb/s, which was why I could never understand the prevalence of advice to "just do your disk backup over SSH". The same disks Samba between Windows and Linux achieve 70-90 mb/s, and switching to HPN-SSH (in either Cygwin or Linux) gives me the same sort of speeds with the AES cipher or none cipher.

These are not slow machines - i7's and i5's.

I've never been able to find a compelling explanation as to how this could be when other network protocols can manage it just fine.

(I wanted to write 50MB =~ 400Mb earlier).

Something in your setup is broken. I get 50MB/s speeds when Linux is on both ends, and even when Cygwin is on one end.

Because SSH requires several seconds to initiate a session, even on a local LAN.

Does anyone know why this is the case? Its always baffled me.

A delay of several seconds every time when connecting via ssh is usually due to the remote host trying to look up your IP address and timing out. Change the remote host's sshd_config file to include the line "UseDNS no" and connecting will be much quicker from then on.

You may be experiencing the GSSAPIAuthentication delay.

The -v flag will help identify where the connection is being delayed.

ssh -v [user@]hostname [command]

Try connecting with the option disabled.

ssh -o GSSAPIAuthentication=no [user@]hostname [command]

You can globally disable GSSAPIAuthentication in ssh config:

echo 'GSSAPIAuthentication=no' >> ~/.ssh/config

Thats probably because the server is trying to lookup the client on DNS: http://www.turnkeylinux.org/blog/slow-ssh

Initiating an SSH connection takes 300ms on my LAN.

If it's taking you several seconds then you probably have a reverse DNS problem. (By default, the SSH server looks up the reverse DNS of the client when it connects.)

It shouldn't take much longer than an HTTPS connection, could be something with your server configuration or latency?

Slow SSH session is usually a DNS issue.

There are many possible causes, but the biggest one is that openssl is notoriously slow at the handshake process. Changing the cipher, playing with DNS, turning off unnecessary features and tuning the order of operations can speed up the process. Otherwise, try a different ssh daemon that doesn't use openssl.

TLS also takes some time to initiate a session, but that hasn't turned anyone off SPDY, since it supports concurrency from then onwards. :-)

It could also be your shell on the remote host taking it's time. Some distros have gotten a bit bloated.

Key management is pretty primitive. It would be nice if SSH integrated better with PGP/GPG. I recently spent too much time messing with monkeysphere, keychain, gpg-agent, gpgsm etc. trying to use GPG derived keys for SSH. While I could cobble something together I didn't feel that it was a solution I could recommend to others as a general "best practice" because it involved too much installing and configuring and was still somewhat brittle (eg. providing key names to gpg-agent in my .bashrc file).

But SSH keys really don't matter.

In PGP/GPG, key persistence matters because you're using them to decrypt messages. Long after content was created you may need the key to decrypt it.

For SSH, the key is only strictly necessary during the session. Key distribution* (of your public key to systems you need access to) is a bit of a pain, but between having your private key(s) where you need them, and authorized keys on various servers, there's not all that much to worry about. Host keys, perhaps, if you want to be rigorous about security.

I'd love to see a project to make this suck less.

It's not clear to me how SSH differs from SSL/TLS conceptually. It seems to me both achieve similar goals (encrypted tunnel, client/server authentication). Perhaps we should take the best bits of both protocols and create a new one? But then, I am reminded of http://xkcd.com/927/.

SSH is designed to serve a single service on a single host. It distributes its host key on the first connection and caches it indefinitely, assuming it will never change. SSH is designed with a limited set of protocol features, and everything else is kind of hacked on top of proprietary client/server pairs. SSH is designed as a loose encrypted session (kind of like a pipe) for an application on a host.

TLS is designed to serve multiple services on multiple hosts. It depends on your browser trusting an intermediary host which validates the host key, so (in theory) the initial connection can't be MITM'd, and so the key can change at any time or there can be multiple keys (which is needed for hosting multiple services on multiple hosts). TLS is designed to integrate tightly into an application.

When you compare the two protocols, TLS is clearly superior to SSH. But in terms of the features they support (tunneling, authentication, etc), it's up to the server to add missing features outside of the protocol to provide for what the client wants to do.

For example, the SSH protocol basically provides an encrypted connection through which you can do whatever you want, similar to TLS. To do IP tunneling with SSH the application server activates extra functionality to connect the encrypted session to a driver which opens an IP tunnel. Or to authenticate your ssh session against a kerberos server, the ssh server does the actual kerberos authentication; the protocol just informs the client of what 'basic' methods they can use, and the client tries to use one that works with the server's methods.

Incidentally, TLS the protocol supports client certificate authentication, which provides similar functionality to SSH's public keys. The HTTP protocol also does certificate pinning.

Maybe it would be nice then if TLS would somehow cache host keys, too? Like my browser caching the relationship:

www.bank.com is with pub key XYZ

Or is this already implemented and I am too stupid to find it?

Since this kind of goes against the point of using an intermediary to verify the host key, and multiple services and hosts make this much more difficult to support, it's not built into TLS. But the application can add support for it. There are experimental web standards and methods for various OSes/applications here https://www.owasp.org/index.php/Certificate_and_Public_Key_P...

I wonder if his chat server has been hit by a botnet yet trying to ssh in and then sending tons of shell commands

Lots of people spamming/flooding/DoS'ing "for fun", but no clueless bots stumbling in accidentally yet.

Btw previous HN thread about ssh-chat here: https://news.ycombinator.com/item?id=8743374

That would be very interesting to watch (albeit a bit annoying for the people trying to chat)

It was difficult for me to get past the self-referential (and reverential) tone of the writing, which felt more like an advertisement for the author's cleverness than a real discussion of the pros and cons of SSH, but it could be I woke up on the wrong side of the bed this morning.

Somewhat relevant: I think Mosh solves a few problems that SSH lacks. This article: https://news.ycombinator.com/item?id=8252093 made it on Hacker News a bit ago. The video is wort the watch imo.

SSH can really be used for almost everything. It's a different thing if it actually _should_ be used for everything.

My favourite one: http://en.wikipedia.org/wiki/SSHFS

Whenever I'm developing for my mobile phone I actually have the contents mounted on my desktop via sshfs as an actual filesystem.

What that means? Don't bother with FTP servers. ssh access to your server is all you need.

If you rephrase this question like this, the answer is evident:

"Why aren't we using a protocol designed to add encryption to pseudo-devices emulating a real text terminal device [1] for everything?"

1: http://en.wikipedia.org/wiki/Pseudo_terminal

Would be great to do this for mosh - mosh-chat It improves over SSH by handling intermittent network connections.

mosh isn't a data stream tool like ssh though, it's actually more like VNC -- the reason it performs better is that it sends snapshots of the terminal over UDP, allowing random packets to be dropped

What surprises me is the lack of total CLI control, potentially through SSH and ideally using keys, of your hosting provider's control panel. Whether it's CLI or some form of TUI, it's bound to be faster and more convenient for many developers.

Cloud providers nowadays usually have some sort of CLI clients. OpenStack comes with a full suite[1]. Google rolled out their own for Google Cloud[2]. And if there's nothing official, there are often tools built by users available for interacting with the API.

Authentication is based on some sort of shared secret rather than keys though, yes.

[1] http://docs.openstack.org/user-guide/content/ch_cli.html

[2] https://cloud.google.com/sdk/gcloud/

Interesting idea, but isn't ssh very sluggish when it comes to throughput and latency? Doesn't it cost quite a bit more in CPU? I'd hate to make ssh the protocol replacement for http for a busy site.

It depends on the clipher you're using and the implementation/hardware.

It depends more on the TCP window size; OpenSSH uses a small one by default and it cannot be changed.

See also http://www.psc.edu/index.php/hpn-ssh

MOre interesting question: Why aren't we using SSH-style authentication for everything? And although they can be used with SSH, I do not mean certificates.

I don't see why client side certificate authentication is excluded? You can easily self sign and create your own CA. There aren't really any downsides. Putty with cryptoapi support exists.

What the...? Does the author just not know anything about SSH, or web browsers? Why would we use SSH for everything?

On top of the fact that they're entirely different protocols and tools designed for entirely different purposes, browsers already support virtually everything SSH does. File transfers, authentication, client certs, multiplexing, key pinning, etc. There is no need to use SSH, and if you did, it would be slower, less secure, and generally more annoying than using the existing tools built into browsers.

I find it aggravating when users read the manual to some software and think they have discovered fire.

What does this even mean?

Your browser supports those things because it implements protocols, like HTTP, TLS, and others. Your browser, or other tools, could support SSH, which I think is the point of the article.

This means the author doesn't know what they're talking about, and is asking a dumb question.

The article's title is "Why aren't we using SSH for everything?", not "Why don't browsers support SSH?". Both make no sense. It's like asking why FTP clients don't support Voice-over-IP.

I read somewhere that SSH can be MITM'ed by a global adversary on the first visit (before it establishes the secure connection). Is that true?

Yes. It does not have any centralized certificate system like HTTPS so unless you can manually verify the host's public key, you will not know whether your first visit is being proxied. Of course, if the first one is proxied, so may subsequent ones, and you would only get a warning if the proxy was removed or if it's key changed.

SSH supports CA-style key signing, and it also supports server fingerprint validation over DNSSEC (search for SSHFP DNS).

Unfortunately neither of these things are commonly used yet. Cloudflare is adding DNSSEC support soon, so hopefully that will change.

It's going to take a lot more than Cloudflare adding DNSSEC support to make SSHFP records viable. Every system running an SSH client will need to run its own validating resolver. If you leave validation to an upstream server you lose a significant amount of security.

Not to mention that if your adversary is the Global Adversary, DNSSEC is mostly useless.

You get shown a key fingerprint at first connection. If you verify that with a secure source you can confirm the host you are communicating with.

Actually bothering and having a secure way to verify the fingerprint are exercises left to the reader.

If you don't actually check the fingerprint of the key matches the fingerprint you were given (in some other secure way, such as walking over to the server), then yes, a first-visit MITM is possible.

Yes. SSH can be silently mitm'd on the first connection due to not having the host key cached.

On the other hand, every web browser that visits a site for the first time (unless it was pinned in the browser you downloaded) attempts HTTP before HTTPS, and is thus vulnerable to all sorts of attacks. All non-pinned HTTPS connections at any time can be mitm'd by a global adversary that generates a cert using a CA your browser trusts.

So technically SSL is much easier to catch being mitm'd, since you only need to worry about the first visit.

Yes, but only on the first visit. In comparison, TLS could be MITM'ed in each individual visit by a localized adversary as long as their location includes a CA you trust - which appears to be the case for every intelligence agency out there.

Which threat model you prefer to deal with is up to you.

It can be MITM-ed on any connection. Whether you approve the host fingerprint is, and has always been, entirely up to you.

Default configuration on all of the system's I have used (mostly Ubuntu) is that you cannot just approve it. You have to actually remove it from the known hosts table, in effect starting a new "first session". You can set it to allow override without deleting the entry though and some system may have this as the default.

Which is still irrelevant: in the default configuration the server I'm connecting to is probably not compromised, I've probably just done something to change keys or routes or names or whatever.

The problem is further back: the default configuration doesn't make it easy to avoid these problems in the first place. If I'm logged in and do something which will change SSH host keys or the like, then the default needs to provide a way for me to make that information easily available globally to other clients which might be aware of this system.

I don't even know of a client that implements this, but SSFP DNS records are the way to do this. http://www.openssh.com/txt/rfc4255.txt

A security conscious client can manually verify a host's SSHFP with a dig record. You'll also want DNSSEC in place to ensure the dns hasn't been spoofed either.

You can fix this with SSHFP records and DNSSEC.

> Why aren’t we using SSH for everything?

Because it doesn't support virtual hosts. And I can't afford 30 IPs for my server.

Otherwise, it's a great protocol.

Couldn't you use the same IP and different ports? There's nothing magic about port 22 - and in fact on production servers you should almost certainly change 22 to something else.

If you believe Jacob Appelbaum, we probably should not be using SSH for anything http://media.ccc.de/browse/congress/2014/31c3_-_6258_-_en_-_...

If you believe that being able to sometimes compromise some implementations of SSH under some circumstances means that you shouldn't use SSH for anything then maybe you shouldn't use the Internet at all.

Yea I was fairly confused about that announcement as I couldn't find anything damning in the released docs about SSH. Do you have a reference to a specific document/slide?

Spiegel briefly touched on it too;

The NSA also has a program with which it claims it can sometimes decrypt the Secure Shell protocol (SSH). This is typically used by systems administrators to log into employees' computers remotely, largely for use in the infrastructure of businesses, core Internet routers and other similarly important systems. The NSA combines the data collected in this manner with other information to leverage access to important systems of interest.

(source: http://www.spiegel.de/international/germany/inside-the-nsa-s... )

Still incredibly vague. If they're archiving all traffic in hopes of decrypting it some day though, it's safe to say we should treat anything on the internet as the shiny side of one way glass.

My guess is that, at least, some router ssh implementation is insecure, possibly not by accident.

From the slides:

Page 19: "SSH [...] Potentially recover user names and passwords" Page 36: "SSH - often have router configurations and user credentials [...]"


Right, I saw that also. Is it referring to routers that happen to run SSH with default root/admin passwords or something? I couldn't find anything more concrete.

The slides say they can sometimes decrypt SSH sessions (with ssmintm?) and catpute usernames and passwords from SSH servers.

The only unexpected thing I could see there was the lack of anything I didn't know how to do...

Yea, sshmitm makes sense also, but I wouldn't consider that a 0day exactly. :P

Relevant bit starts around 00:25:30

Also: https://twitter.com/ioerror/status/549327936361611264

This link 404's for me; can you give a summary?

That's weird all the CCC videos were returning 404. The link appears to be working again now. It's a few days since I watched but I inferred that SSH may have some undisclosed vulnerabilities.

There is something about Finnish products that make them very strong, Nokia, SSH, Rovio.

Because schools and filtering software

Sorry the title has been revised to "Why aren’t we using SSH for everything?" in case mods see this.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact