

Ask HN: I need to transfer 10TB of Data across Pacific - stkim1

I just have no idea what to use. Would there be anything better than carrying ten 1TB HDD myself?
======
cat9
The best solution depends entirely on your optimization conditions.

If transaction time is important, hard drives are high bandwidth. Ten 1TB hard
drives are $500. Two 6TB hard drives are also $500, but you have fewer drives
to worry about shipping. Five 2TB hard drives are about $370, if drive price
is more important. Using ten 1TB drives is only best if we're optimizing for
load / unload time and have that degree of parallelism in the pipeline.
Otherwise, one of the other two will be better, depending on price sensitivity
vs. other factors.

If it's not as time-sensitive, something like BitTorrent Sync will get the job
done. This doesn't require shipments, additional hardware, or mucking about
with hardware at either end, so it's very nice if "get it there ASAP" isn't a
constraint.

Regardless, don't ship the only copy.

~~~
stkim1
Thanks for the concise summary.

------
bogrollben
Mail it to me at the hacker house in Kansas City. We do uploads for people all
the time using Google Fiber. Probably take a few hours (YMMV). Google Homes
for Hackers. My cell is right there on the site.

~~~
namecast
+1 for being awesome - but the OP mentioned 'across the Pacific' \- if he's
got to transfer 10TB to, say, Sydney or Beijing, I'd bet your upstream won't
be the limiting factor, it'll be the congested peering ports and high latency
links between the States and APAC providers.

That said, nothing ventured, nothing gained. Maybe try a speed test between
the hacker house and the intended AS before anyone ships a drive anywhere, and
see if the other end can keep up with you?

------
lazylizard
[http://filecatalyst.com/](http://filecatalyst.com/)
[http://filecatalyst.com/open-source-fast-file-
transfers/](http://filecatalyst.com/open-source-fast-file-transfers/)
[http://www.jscape.com/blog/bid/78782/How-to-Boost-File-
Trans...](http://www.jscape.com/blog/bid/78782/How-to-Boost-File-Transfers-
Speeds-100x) [http://asperasoft.com/](http://asperasoft.com/)
[http://www.signiant.com/](http://www.signiant.com/)

------
tantalor
[https://what-if.xkcd.com/31/](https://what-if.xkcd.com/31/)

~~~
stkim1
See? this is funny. Some says this is the age of data. 10T is just a tiny
amount of bits for some entities, and I cannot reliably transmit that much
across pacific other than shipping actual HDD. :(

Thank you for the comic.

------
toast0
Two 6 TB or three 4 TB is probably a better choice than ten 1 TB. Although you
may want to have an extra disk so one could totally fail in transit; so maybe
six 2 TB. Build a big parity archive (par2 or similar) first. If you have
1gbps end to end, it would take most of a day (make sure you have big TCP
windows)

------
laurencerowe
With long ping times you want to split your upload into many parallel parts so
the odd dropped packet doesn't bring things to a halt. Then it just depends on
your bandwidth.

We were able to saturate the gigabit nic on an ec2 instance in Ireland
uploading to S3 in Oregon. At that rate a 10TB will take a little over a day.

~~~
georgerobinson
I'm puzzled. Can you explain what you mean by bringing to a halt?

Do you mean a dropped packet might terminate the TCP connection; or at worst
result in the timeout of a segment and a return to slow start? Nevertheless, I
would have thought a timeout is a rare event at best, even on high latency
links, due to the post-Jacobsen RTT estimator and fast retransmit kicking in
at 3 duplicate ACKs?

------
gesman
5 x 2TB HD's bought at Costco + shipping charges.

Total cost: <$1k

Rather small package. "Upload" time <10 days.

------
codeonfire
Whether you can use distributed web services like s3, elastic mapreduce, or
google storage to copy it largely depends on what kind of ISP connection you
have on both ends. With gigabit and multipart transfers it seems feasible
within a week.

------
drallison
Sneakernet
([https://en.wikipedia.org/wiki/Sneakernet](https://en.wikipedia.org/wiki/Sneakernet))
is likely the best choice.

------
dgomesbr
Bit torrent it to the other side of the pacific. With a 35mbps connection will
take like a month.

------
meat_fist
Mail a HD.

------
001sky
FedEx or similar

------
deskinner
Globus.

