
A Desperate Plea for a Free Software Alternative to Aspera (2018) - fanf2
https://www.ccdatalab.org/blog/a-desperate-plea-for-a-free-software-alternative-to-aspera
======
ncmncm
Ex Aspera Dev here. I did the encryption and early parallel work.

There is a lot of good science behind fasp. An advantage it has over IETF
protocols is that both ends trust one another. Another advantage, until
recently, was out-of-order delivery.

The protocol totally ignores drops, for flow control. Instead, it measures
change in transit time. The receiver knows, the sender needs to know, but the
useful lifetime of the measurement is less than the transit time. This should
make an engineer think "control theory!", and did. So, the receiver reports a
stream of transit time samples back to the sender, which feeds them into a
predictor, which controls transmission rate. Simple, in principle, but the
wide Internet is full of surprises.

If you think this wouldn't be able to go a thousand times faster than TCP, you
have never tried moving a file to China or India over TCP. :-) (Customers used
to report 5% drop rates.) Drops and high RTT are devastating to traditional
TCP throughput on high-packet-rate routes; read about "slow-start" sometime,
and do the math. Problem is that for untrusted peers, drops are the only
trustworthy signal of congestion. Recent improvements where routers tag
packets to say "I was really, really tempted to drop this!" help some.

Torrents get the out-of-order delivery and the lower sensitivity to drops, but
its blocks are too big.. Others commented that opening lots of connections
gets around some TCP bottlenecks, but that helps much only when the drop rate
isn't too high (i.e. not to India or China).

~~~
eps
Rate control sounds nearly identical to TCP Vegas then, no?

But out of order delivery and (I'm guessing) out of order retransmits appear
to be unique to fasp. What did you mean about these being unique "until
recently"? [edit - nevermind, saw below you were referring to sack]

PS. I remember looking at fasp ~10 years ago and it looked like a fantastic
tech. Someone's taking things that many people only talked about and putting
them into a product that really worked well. Cutting edge stuff. A job like -
a dream for many :)

~~~
ncmncm
No, quite different from Vegas, if the graphs are to be believed. What I would
see on real trans-oceanic links was a sub-second ramp up to a flat, stable
rate, and stepwise rate changes during the session as other flows started and
stopped, with no visible oscillation.

(I am looking at [https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-
block/](https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/) .)

Fasp does not rely on RTT ACK timing, which is very noisy, but only outgoing
delays.

The routers are applying their own algorithms to simulate sane queue timing,
but they are doing all kinds of crazy shit under the hood to get better link
performance. The queues, IOW, are a fiction provided to the endpoints to
simplify their job and generate understandable, thus tunable full-route
dynamics.

But the ACKs get completely different treatment, because traditionally nothing
cared about ACK timing.

So, "onset of queuing", e.g., is pretty meaningless in practice, and fasp
makes no attempt to detect it.

------
phonon
Have you tried [https://github.com/openrq-
team/OpenRQ](https://github.com/openrq-team/OpenRQ) or
[https://github.com/harmony-one/go-raptorq](https://github.com/harmony-one/go-
raptorq) or
[https://github.com/cberner/raptorq](https://github.com/cberner/raptorq) (the
last seems to have excellent decoding/encoding performance)?

(Raptor codes (a type of FEC) can essentially transfer over UDP at the
_underlying line rate, even if packet drop is high_. In other words, if it is
a 1 gbit/s link, with 5% packet loss, you will be able to send over 900
mbits/s without needing any TCP features). This is also helps high latency
scenarios, as you don't need test/back off how much data to send...you just
send as much Raptor encoded data as you want down the pipe, and as long as you
send enough recovery packets with it, you will be able to reconstruct the
original flawlessly).

(Also see
[https://par.nsf.gov/servlets/purl/10066600](https://par.nsf.gov/servlets/purl/10066600)

[http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf](http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf)

[https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ug...](https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ugd/4c5b11_93bc499bed254bf29057bfb5f8c44867.pdf)

[https://youtu.be/bYPbat-FFTo](https://youtu.be/bYPbat-FFTo)

[https://github.com/mk-fg/python-libraptorq](https://github.com/mk-fg/python-
libraptorq))

~~~
Scaevolus
You don't need 5% packet loss on a high latency 1Gbps link to trash
performance. Even .1% packet loss is enough to cut throughput massively on
long fat networks (LFNs).

Erasure coding helps, but it demands much more CPU than simply having a very
wide window for retransmits. For bulk data transfer of files, retransmitting
any part of a 1GB window is trivial.

~~~
traverseda
You mean with raptorQ 5% loss trashes performance by more than 5%?

~~~
phonon
No, the overhead is fairly minimal. A 5% packet loss (which is very bad) on a
gigabit link where the sender is thousands of miles away might have an
effective throughput of 930 mbits per second with RaptorQ. With regular TCP
you would get only a few megabits/s, as it would back down until the loss rate
was nearly zero.

------
ovidiul
This looks interesting as well as it's developed at CERN and uses plain tcp
protocol [https://github.com/fast-data-transfer/fdt](https://github.com/fast-
data-transfer/fdt)

~~~
YarickR2
Second this. When we needed to deal with somewhat similar issues (quickly
transfer multi-gigabyte archives over massively unreliable link) , we ended up
with FDT ; it can be easily scripted , and transfers data between two hosts
(without using a network of torrent peers) at that hosts' link speed. Used it
to transfer builds between US and Russia, and Russia and China .

~~~
branzo
+1

Used fdt to transfer a 6TB archive out of AWS very smoothly at full speed.

------
justinsaccount
The 'proof' shown that aspera is 200x faster shows an ETA of 2:15 compared to
00:09. First of all, this is 15x faster, not 200.

I tried downloading the file using axel -n 32, and it took 1:28, which is only
9x slower. Curious though, when it started out with all threads running it was
transferring data at 29594.2KB/s and hit 50% after only a few seconds, but by
the end with only a few threads left running it was only doing 5625.8KB/s.

It looks like the performance varies a lot, possibly due to multiple
interfaces or wan links being used.. or issues with their server.

Using axel -n 64 over http I was able to fetch it in 50s, which is only 5x
slower, and most of that time was spent from 98 to 100% as the last few
connections finally finished.

A smarter client that more aggressively re-fetched chunks being downloaded
over slow connections would likely match the same performance.

~~~
StavrosK
Yeah, and one window shows speeds in MB/s and the other in Mbps (a 10x
difference in units right there).

~~~
justinsaccount
ah, I bet they missed that which is how they ended up with 200x instead of
under 20x

------
KaiserPro
Ex Aspera user here.

I worked in VFX, so I've been using aspera since before it was owned by IBM.

Depending on what line speed you have, but for a gig link we made a simple
protocol using parallel TCP streams.

Basically, it chunked up the large file into configurable sized chunks, and
assigned a chunk to each stream.

Another stream passed the metadata.

This has all the advantage of TCP, with less of the drawbacks of a custom UDP
protocol.

For transferring files from london to SF we were getting 800mbit/s over a 1
gig link.

------
grizzles
Why not use torrents over wireguard? From an engineering perspective I'd
personally be hard pressed to come up with a more optimal solution for private
big data transfer.

If Aspera is as good as the parent suggests it's probably because they are
operating a bittorrent-like network of geo-distributed peers. It would be very
cheap to emulate that using cloud computing providers, especially if you are
discarding the data after transfer.

~~~
KaiserPro
Aspera uses multiple UDP streams to push data.

The special sauce is measuring the packet loss quickly enough to make sure
your not over saturating the link

Apart from that, its a fairly simple protocol.

~~~
ncmncm
Aspera only uses multiple UDP streams if the line rate is too fast for one
sending process to keep up. Very often the actual bottleneck is the receiving
storage backend.

It absolutely does not measure packet loss at all. The only response to a
dropped packet is to request a re-send.

The protocol is simple but depends on very smart sending rate control. Much of
the complexity of the current version is to perform well when the receiver has
a slow disk, or the link goes through a satellite.

------
john_moscow
Honestly speaking, a good way to solve this is not trivial. You would need to:

1\. Measure the existing performance on specific data sets.

2\. Understand how many of the bottlenecks come from the network latency vs.
I/O limits vs. CPU bottlenecks (if using compression).

3\. See if any domain-specific compression is needed.

4\. Document the typical use cases. Understand why the current solution sucks
(e.g. requires redundant user actions). Write down user interaction scenarios.
Design the UI to be as efficient as possible for those scenarios.

This is a non-trivial amount of work that would require a lot of back-and-
forth interaction and on-the-go requirement changes and I don't think it's
entirely honest to ask someone to do this work for free in the name of cancer
research (after all, you are not donating most of your paycheck to charities,
are you?). If the existing solution by IBM sucks, how about making a Request
For Proposal [0] and seeing if smaller software vendors could offer something
better given that you are actually willing to pay for the work?

P.S. A student/hobbyist can probably whip out some sort of a parallel TCP-like
thing with a large window for free, but you would get the same performance by
just cranking up the TCP window size via sysctl and using multiple HTTP
threads (htcat was suggested earlier in the comments).

[0]
[https://en.wikipedia.org/wiki/Request_for_proposal](https://en.wikipedia.org/wiki/Request_for_proposal)

------
yoz-y
Way back when I had a problem on my network that was capping the FTP/SFTP
speeds to about a fifth of my download speed. Torrents worked fine. To fix my
problem I have cobbled up an extremely hacky solution which consisted of
opening several listening sockets, ssh-ing to the remote machine and net-
catting the file by chunks in several processes in parallel. It actually
worked. The code is here [https://github.com/jlegeny/blazing-
speed](https://github.com/jlegeny/blazing-speed) more for archiving purposes
than to be used by anybody for anything.

------
mattbillenstein
I benchmarked Aspera against alternatives for getting data from on-prem
storage into the cloud (google cloud storage) a year or two ago on a 10Gbps
link.

Aspera was faster on a single stream, but if you have a lot of files to move
around (you usually do) you can just multiplex a bunch of tcp streams to get
the same throughput.

So like in the example on their page, if you have 20 wget's hitting their ftp
and there are no other bottlenecks, the throughput will be similar to using
Aspera - and you have just a greater variety of free tooling for tcp based
those protocols...

~~~
ncmncm
Sending on lots of TCP streams can get you to 80% line utilization if the drop
rate is good and low. That's often good enough. On some lines, with the
"wrong" number of TCP connections, the rate will oscillate, and you will be
lucky to get 50%. Tuning that is a black art some people enjoy.

~~~
mattbillenstein
Yeah, I was probably working in an environment with pretty low packet loss.

Looking at the problem holistically - if you're going to spend money on Aspera
to fix situations with high packet loss, maybe you're better off spending
money on a higher quality connection and using tcp.

------
01CGAT
FTR, they say: "As you can see, the ascp client performs over 200x faster than
FTP, and can download the whole file in 9 seconds. It’s pretty magical." But
the example shows MB/s vs Mb/s. It's not over 200x times faster, but about 17
times faster.

------
zxcvbn4038
I feel the author’s pain - Aspera is a clunky 90s styled Ruby on Rails app
that is about the most cloud unfriendly piece of software I’ve encountered,
I’d love an alternative.

I honestly don’t care about about their proprietary UDP protocol, it’s nothing
special, just another way to copy bits onto a wire. Dime a dozen.

The true value of Aspera is they provide an integrated browser plug-in that
lets the technically challenged reliably upload large files. If the transfer
is interrupted or either side changes addresses it deals with it gracefully.
It’s also a bridge to AWS S3.

I spent some time looking for a replacement and while there are numerous
download managers that facilitate people downloading large files, I couldn’t
find any upload managers with the same level of integration and polish.

About the closet thing I could find is Cyberduck, but it’s not integrated with
the web browser, not as easy for technically challenged people to use, and
there is no support (community support but seems really hit and miss). However
it does make good use of Amazon’s multiple upload api and will happily fill
whatever wire it’s connected to.

Torrent software has largely the same pros and cons Cyberduck does.

------
dgemm
I have seen this software before. I suppose the speed boost results from using
multiple parallel connections, meaning it could make better use of aggregate
links and multipath networks? Is there anything else to it?

~~~
mattrp
It does what it says and it’s expensive. There’s a couple of commercial
packages like this - two that come to mind are vcinity and Signiant. A
solution can be made but every time I look at the market you find very few
customers with a real need to move terabytes that can’t solve their problem
with a snowball-style solution. Those that do usually also have the economics
to justify a commercial package.

~~~
ncmncm
Don't know about vcinity, but Signiant was always a marketing-heavy,
engineering-light organization. Their focus was on user convenience features,
and they almost ignored actual transit engineering.

------
dboreham
A proprietary protocol that somehow facilities throughout orders of magnitude
than non proprietary alternatives. I call BS.

~~~
sansnomme
If they are like Riot and have their own fiber, it is not unlikely. There are
tons of boring optimizations open source software also won't bother with
without significant sponsorship (some could be specific to a particular file
format). If you look at most network (and also video) protocols, they are
accumulations of decades of improvements rather than neural network style
overnight breakthroughs.

~~~
big_chungus
Do you know of any resources where I can learn more about these boring
optimizations?

~~~
SaxonRobber
MIT open courseware has a course on Performance Engineering which details some
of the types of low level optimizations that can be done. It requires a lot of
measuring and problem specific solutions.

~~~
brians
I’ve helped with that course in the past. It’s awesome. It’s completely
unrelated to WAN networking. I found the IETF groups working on these problems
more helpful.

~~~
big_chungus
Thanks. Any specific recommendations for someone fairly new to low-level
networking? IETF docs can be a little dense :)

------
danbmil99
Question at the meta-level: what software/platforms are producing, analyzing,
modifying & consuming this data?

Is it possible some legacy systems that produce and consume these massive
files on-site would more sensibly run in the cloud, directly & selectively
accessing the data chunks they need over fast backbone connections?

Also, is there any room for an rsync type approach, sending compressed deltas
rather than naively sending huge files that may be redundant?

Not to say disintermediating a BigCO expensive patented vendor-locked-in MLPOS
(Market Leading Piece Of Shit) doesn't sound exciting -- it does. I'm just
curious and a bit skeptical that it is always necessary to mass-copy all this
data over and over.

~~~
rcthompson
The systems that produce massive DNA sequence files are machines reading the
sequences of actual physical DNA molecules. The cloud can't sequence DNA, as
far as I know.

~~~
ncmncm
Most DNA sequences are only slightly different from sequences you already
have. There ought to be mature differs by now that would enable you to send
just differences. If not, there is a huge opportunity for somebody.

~~~
rcthompson
Sure, you can compress the sequences by diffing against the published genome
sequence, and many tools exist to do just that. But that alone maxes out at
50% compression, because the other half of the file is the quality scores that
quantify the confidence level of each base call, for which there is no
reference. Of course there are patterns in the quality scores, and you can
compress these as well using the usual techniques for general compression, but
it's going to limit how much you can compress without information loss. In
addition, the sequencer assigns an unique ID to each sequence read. It's often
ok to discard this information, but occasionally you need it in order to trace
a specific problematic sequence through the entire analysis pipeline. So if
you want lossless compression, you have to compress these as well. They should
be rather compressible, since they are typically a prefix global to the file
followed by a number that increments by 1 for each sequence. But for a general
compressor you can't just automatically _assume_ that, because according to
the (lack of a) file format spec, each ID could be anything as long as they're
unique within the file. So if you want guaranteed lossless compression of any
FASTQ file, your compressor needs to be sufficiently general.

Beyond that, compression of sequences by encoding only the differences from a
specific reference sequence somewhat ties you down to using a specific
reference, since that reference sequence is required for decompression. This
is inconvenient because the reference sequence for a species is updated over
time, and if you want to use the new reference, you'll need to decompress with
the old reference and then re-compress with the new one. Do you want to do
that for all your data, every time the reference is updated? Reference-based
compression also means the compressed files are no longer self-contained,
which may not be acceptable for certain use cases.

None of these issues are fundamentally impossible to address, but the point is
that it's not nearly as simple as it sounds, and anything fancy you do to try
and compress a sequence file will generally impose some additional
requirements on what the receiver needs to do in order to read the file. And
many receivers are not very technically inclined researchers whose plates are
already full of other things they need to be doing besides figuring out how to
decompress your new unfamiliar sequence compression format,

~~~
ncmncm
Thank you for this enlightening explanation. I should have guessed that it,
like most things, is not as easy as it looks.

------
ewwhite
I've used UDT via UDR for rsync optimization and transfers over high-latency
links: [https://github.com/LabAdvComp/UDR](https://github.com/LabAdvComp/UDR)

~~~
patelh
Nice! Thanks for the info.

------
aamargulies
Try looking at bbcp perhaps?

[http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm](http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm)

~~~
dnautics
bbcp doesn't work so well for some reason. Where I used to work we had a
senior dev that claimed that he could just repurpose bbcp for this. Then he
gave up and wrote something custom in c++. Then he rewrote it in go, but used
(among other things) non-multithreadsafe primitives, then when asked to make
it encrypted, he used epoll with tls (which is apparently not a thing in go? I
don't know). I told him he should just use DTLS or hell even a one-time-pad
encrypted UDP stream with backpressure management, but he didn't listen to me.
Then he rewrote it in C++ again and then went back to the go version. When I
left the company, it still wasn't working.

Also, this senior dev never wrote unit tests.

On the other hand, maybe bbcp will work, and the senior dev just didn't know
what he was doing. There was another senior dev who, seeing what was coming
down the pike, left his job, and on his way out he was like, "yeah you can do
it with bbcp, just you gotta tweak your tcp congestion rules on all the hops
(which we _could_ do, but is probably not an option for OP)"

~~~
na85
>On the other hand, maybe bbcp will work, and the senior dev just didn't know
what he was doing

I mean... based on your description it sure sounds like he didn't know what he
was doing.

~~~
dnautics
Well yeah but the competent dev would have told you that it wouldn't work for
OP.

------
TNick
In the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?

~~~
fanf2
Yes, I got a similar list in response to my link on Twitter
[https://twitter.com/fanf/status/1210858696558944256](https://twitter.com/fanf/status/1210858696558944256)

But it looks like GridFTP and Globus are basically dead
[https://opensciencegrid.org/technology/policy/gridftp-gsi-
mi...](https://opensciencegrid.org/technology/policy/gridftp-gsi-migration/)

Tsunami sounds good but maybe needs some updating? I haven’t looked closely...

~~~
reidacdc
Globus is definitely not dead[1], it has an active community with strong
uptake in the Materials Science world (where I work), among others. The Globus
folks have a repo for many popular Linux distros, it's reasonably
straightforward to get started. (I'm a user of it, not a developer or
necessarily an advocate.)

It's not free for "managed" (Enterprise) applications, though, which is what
the OP seems to be looking for, and I'm not sure it's the best choice for
high-speed.

Something that I think _is_ free is the CERN FTS [2], which can use a GridFTP
back-end, so possibly you can roll your own high-speed big-data infrastructure
that way.

[1] [https://www.globus.org](https://www.globus.org) [2]
[https://fts.web.cern.ch/](https://fts.web.cern.ch/)

~~~
CaliforniaKarl
I concur that Globus is definitely not dead! We're transferring 10s to 100s of
TB per month with it. Globus is continuing to maintain GridFTP for this
purpose, as part of Globus Connect Server and Globus Connect Personal, and it
will happily saturate a multi-Gbps link.

And personally, I'd like to see more people getting a Globus subscription.
It's cost-effective when compared to tools like Aspera, and helps fund the
development of Globus software and features.

------
saagarjha
> Aspera is owned by IBM - does their client spy on what else is running on my
> system and report it back to their headquarters as business intelligence? If
> asked, would they share this information with the government? Without the
> source code, it’s impossible to tell!

I mean, with some reverse engineering, mentioned later in the post, it’s
really not.

------
quasarj
My group is about to start using Aspera as well.. I too wish there was a good
alternative I could suggest to my bosses. And I'm still skeptical of their
claims... but I haven't gotten to test it over an actual bad link yet (over a
good link it does nothing, which a simple test will prove).

~~~
CaliforniaKarl
I don't know what your needs are, but Globus might be a good alternative.

[https://www.globus.org](https://www.globus.org)

It's being run out of UChicago, and I know they have at least one large
company using them. Feel free to reach out to the Globus team, or shoot me an
email!

------
sgt101
There are other wan accelerators out there - like Riverbed.

But, I think Aspera uses some specialized compression for genomics data, and
genomics compression is not easy.

My suggestion would be to talk to the telco that provides your wan. If you are
using a direct internet connection my suggestion would be : don't.

~~~
wtracy
The author is discussing intercontinental data transfers. Not many
organizations have a WAN that can cross oceans without hitting the internet.

~~~
moonbug
almost all HE and research institutions participate in nrens which provide
3xacrly those peerings

~~~
ncmncm
Yes, but that typically takes government cooperation, and you still don't get
better latency -- even if you have neutrinos.

~~~
sgt101
It doesn't, it takes buying access to a link.

Latency isn't the problem here - it's throughput.

~~~
ncmncm
Oh, how nice it would be to live in such a simple world.

Latency turns out to have a very great deal to do with how hard it is to get
the nominal throughput you pay for.

------
binarnosp
What about UDT [1]? Several open source libraries exist already.

[1]: [https://en.wikipedia.org/wiki/UDP-
based_Data_Transfer_Protoc...](https://en.wikipedia.org/wiki/UDP-
based_Data_Transfer_Protocol)

------
edwintorok
Would QUIC or TCP with BBR congestion control help?

~~~
compressedgas
TCP with PCC, even? I don't know how BBR compares with PCC.

Previously
[https://news.ycombinator.com/item?id=8381480](https://news.ycombinator.com/item?id=8381480)
(2014)

------
TheNetEffect
Another commercial alternative is M-Strean - non proprietary TCP based
parallel streaming.

Limited to work with CIFS / SMB / NFS and S3 compatible / OpenStack Swift
compatible / Ceph / Azure storage.

Commercial model is server based with no additional bandwidth charges .

Data stored in a non proprietary fashion so bi-modal access to data possible.

Not sure if it is working with QUIC or not as it is mentioned in blog posts.

Seems heavily focused on the media and entertainment industry.

[https://storagemadeeasy.com/M-Stream/](https://storagemadeeasy.com/M-Stream/)

------
electrum
Facebook open sourced WDT, which seems to have the same goals:
[https://github.com/facebook/wdt](https://github.com/facebook/wdt)

------
telendram
Beyond more profound changes to the underlying transport protocol, bandwidth
acceleration can be provide through a combination with fast compression.

A good setup here would be to pipe data to `tar`, then `zstd`, then
`ssh`/`scp`, and do the reverse on the receiver side.

From an end-user standpoint, all this could be abstracted away if one could
get `wget` (named in the article) to support zstd compression natively, which
is possible since zstd is also a web compression standard.

Such compression support is, by the way, available in `wget2`.

------
gnufx
There's a meta answer about the general approach to such problems: Ask what
other research areas must have tackled it, and then find out what they do. The
obvious answer in this case is HEP, which would lead to some of the tools
mentioned. (There's also [https://www.psc.edu/index.php/hpn-
ssh](https://www.psc.edu/index.php/hpn-ssh) but I don't know how it compares.)

It's unfortunate how many researchers don't do such research...

------
tonyg
I wonder why something like
[https://github.com/htcat/htcat](https://github.com/htcat/htcat) doesn't fit
the bill.

------
TNick
If you think about giving zmq a go, take a look at my library, here [1], [2].

Maybe we can talk about compiling a test to see how it compares?

[1] [https://pypi.org/project/p2p0mq/](https://pypi.org/project/p2p0mq/)

[2]
[https://p2p0mq.readthedocs.io/en/latest/](https://p2p0mq.readthedocs.io/en/latest/)

~~~
Supermancho
ZMQ is too unreliable at transferring data which is a fraction of the stated
use-case. Streaming realtime data, ZMQ had problems dropping data
unpredictably under load testing, so ZMQ never made it out of testing for us.
The protocol is too naive.

------
lazylizard
I think this lists some
alternatives...[https://www.chpc.utah.edu/documentation/data_services.php](https://www.chpc.utah.edu/documentation/data_services.php)

I'm curious. How fast are these compared to netcat/socat/mbuffer?

~~~
tyingq
It mentions rclone with the parallel option of "\--transfers=N". I wonder how
well that does.

------
xxgrinlobs
Try NKN (New Kind of Network) for this task

[https://medium.com/nknetwork/nkn-file-transfer-high-
throughp...](https://medium.com/nknetwork/nkn-file-transfer-high-throughput-
file-transfer-using-nkn-client-21d261d87658)

------
randall
Dat isn't designed for speed, but I think academic data is it's primary goal,
so modifying it for speed seems within its charter.

[https://www.datprotocol.com/](https://www.datprotocol.com/)

------
catalogia
> _We’d have to fly all over the world with suitcases full of harddrives to be
> able to collect our data!_

The postal system or commercial courier services seem like better options than
buying plane tickets..

------
killjoywashere
How about mpscp from Sandia?

[https://www.osti.gov/servlets/purl/1146079](https://www.osti.gov/servlets/purl/1146079)

------
01CGAT
Maybe OpenVPN + UDPspeeder could do something:

[https://github.com/wangyu-/UDPspeeder](https://github.com/wangyu-/UDPspeeder)

------
richardji
I wonder how will aria2 stack up if you use it instead of wget?

[https://aria2.github.io/](https://aria2.github.io/)

------
moonbug
gridftp's the tool to use, and Globus makes it easy.

------
krisoft
Looks like the plea has been up since a while. Does anyone know if they had
any progress on the problem? Is the problem still standing?

------
danbmil99
Btw attempted to contact you at the email you put on your page. Interested in
talking about this project.

------
unixhero
Can they not leverage torrents for this purpose?

------
amelius
Question: if everybody used Aspera (instead of conventional transfer methods),
would that negatively impact internet speeds overall?

