
HPN-SSH: High Performance ssh and scp - dr_linux
http://www.psc.edu/index.php/hpn-ssh
======
FiloSottile
They apply cleanly when building for OSX and work well with the Homebrew
patches infrastructure. Get them with

    
    
        brew install --with-hpn https://raw.githubusercontent.com/FiloSottile/homebrew-dupes/423f9eed0e557f4a63947b04f5dbe22c895ff6f7/openssh.rb
    

Or, once the PR [https://github.com/Homebrew/homebrew-
dupes/pull/358](https://github.com/Homebrew/homebrew-dupes/pull/358) is
merged,

    
    
        brew tap homebrew/dupes
        brew install --with-hpn openssh
    

EDIT: fix, thanks dkubb

~~~
dkubb
Did you mean "brew install --with-hpn openssh"?

------
Dobbs
This is buggy. We had it deployed on a small number of servers , basically our
backup server and a few servers that needed to backup to the backup server.

We would run into problems with standard SSH connecting to HPN-SSH these bugs
were confusing and difficult to debug until we realized that the difference
was HPN-SSH and reverted to standard SSH all problems went away quite quickly.

~~~
rapier1
Could you tell me what sort of problems you were experiencing? I'm more than
happy to take bug reports at hpn-ssh@psc.edu.

------
claudius
What is the license of these patches? There doesn’t seem to be anything on the
website, nor in all of the patches available for download.

Are there other reasons apart from licensing why this has not been merged into
mainline OpenSSH?

Edit: At least the multithreaded AES-CTR cipher has license information in the
header, which seems to be standard BSD. It stands to reason that the rest of
the patches have the same license?

~~~
peterwwillis
The Sourceforge page says the project is BSD-licensed. But if the license
isn't applied to all the files, and no other root-directory file says that all
the files contained within are licensed as such, i'm pretty sure you're not
legally covered. If I were a distro I wouldn't include these without asking
the authors to put a license declaration covering all the patch changes at the
beginning of the file.

~~~
rapier1
I am the author. Honestly, I initially released them without any sort of
license - entirely free from any sort of license restriction. People don't
seem to understand that so I said they have a BSD license on them. Personally,
the license is kind of a moot point (in my view) as the once you patch OpenSSH
with them the code is subsumed by the license used by OpenSSH. The only but
that might be up in the air is the aes-ctr cipher. In any case I'm trying to
find the cycles to include the license and then incorporate it into all of the
bits and then build new packages for sourceforge. In the mean time, take my
word (as the author) regarding the license. It's free and open. Just keep the
attribution. Send me mail at hpn-ssh@psc.edu for identity confirmation if you
like.

------
kolev
Why isn't this merged upstream, I wonder? Here's a thread I found for Debian:
[https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=693424](https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=693424)

~~~
kolev
There's an Ubuntu PPA:
[https://launchpad.net/~w-rouesnel/+archive/ubuntu/openssh-
hp...](https://launchpad.net/~w-rouesnel/+archive/ubuntu/openssh-hpn)

------
noedig
ESNet has a good website for high-performance data transfers:
[http://fasterdata.es.net](http://fasterdata.es.net)

In addition to host and network tuning guides they also have a list of data
transfer tools, including HPN-patched sftp and scp, but they recommend GridFTP
because it supports parallel streams: [http://fasterdata.es.net/data-transfer-
tools/gridftp/](http://fasterdata.es.net/data-transfer-tools/gridftp/)

------
joelio
I've been using this a lot recently -
[https://github.com/LabAdvComp/UDR](https://github.com/LabAdvComp/UDR) \- A
UDT wrapper around rsync - No need to patch _anything_ , therefore no
maintenance woe

------
arbalist
After finding that using scp to copy stuff over my local network (and before
finding out about hpn-ssh) I wrote this icky script to get better performance
copying moving stuff around:
[https://github.com/toofar/fcp](https://github.com/toofar/fcp)

It just wraps ssh+netcat+tar. The ssh connection is just used to setup the
netcat (actually nc) connection like in mosh.

It doesn't have much dependencies (although it can use pv if installed on the
local machine) and so should work on most platforms (didn't work on android
last I checked, but I think I know why). It's a pretty ugly script (in
particular errors are not propagated back to the terminal and backgrounding pv
can lead to offset output) but it works and can even use scp's bash_completion
script for completion on a remote filesystem.

~~~
reacweb
no need of netcat: tar c . | ssh user@host tar x -C destdir

~~~
arbalist
The need to use netcat is that it is faster. It gets better throughput in the
presence of plentiful bandwidth. The whole point was to avoid using ssh for
bulk transport.

~~~
reacweb
ssh user@host "nc -l 1234 | tar x -C dest_dir"

and in another terminal:

tar c . | nc -p host 1234

If you do not care about security, why not using rcp ?

------
auxiliation
Gentoo has this as a USE flag for openssh and is enabled by default. Be aware
that the multithreaded AES-CTR patch has compatibility conflicts with some
libssh versions and possibly other implementations. XBMC and Synology in
particular have both been unable to connect to servers using that patch.

~~~
rapier1
Honestly, I'm not aware of these connection problems. If you have any further
information please send it to hpn-ssh@psc.edu.

------
ominous_prime
I wonder what the current performance difference is to OpenSSH6.1+? I think
those graphs they have might be very old, and there's been a lot of
improvements in the core OpenSSH code since then.

------
dekhn
Native OpenSSH performance seems quite good these days- are you certain this
is still necessary? Either the SSH client and server I used had this patch, or
the windowing code in SSH was removed/modified.

~~~
rapier1
The main problem is that the internal buffer size for OpenSSH is limited to
2MB (actually it's closer to 1.25MB in real world terms). This is pretty good
for a lot of home users. However, HPN-SSH was written for the HPC communuity
where we have have BDPs of well more than that (8MB is not uncommon). So if
you have a BDP of more than 2MB HPN-SSH is still going to be a win for you.

------
fsiefken
2 questions: 1) would this work with mosh? 2) lzo/lz4 has lower latency then
zip, is there an ssh patch supporting this and would it be noticable in a 2k4
scenario?

~~~
Wilya
I don't think mosh-type scenarios are what HPN-SSH aims to improve. It is more
targeted at high-bandwidth, high-latency scp transfers.

(say, 300Mb+ connections with 30ms+ latency)

------
matt-attack
I'm drooling over this. Will this ever show up in RedHat?

~~~
ominous_prime
Not likely. The OpenSSH team is very conservative, and RedHat isn't likely to
make such an invasive change into a core utility.

PSC has been maintaining these patches for a long time. I'm pretty sure I was
using this nearly 10 years ago. If you have a need for this, it's not hard to
build it yourself.

Ah, found the initial release date: July 7, 2004; yeah 10 years.

------
batbomb
check out bbcp if you are interested as well.

~~~
alexw1
Do you mean bbftp?

[http://www.nas.nasa.gov/hecc/support/kb/bbftp_147.html](http://www.nas.nasa.gov/hecc/support/kb/bbftp_147.html)

~~~
kevinbowman
I'm guessing they meant this:

* [http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm](http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm)

* [http://www.slac.stanford.edu/~abh/bbcp/](http://www.slac.stanford.edu/~abh/bbcp/)

* [http://www.slac.stanford.edu/~abh/bbcp/bin/](http://www.slac.stanford.edu/~abh/bbcp/bin/)

I've not tried it, though.

