

Ask HN: How to upload lots of data? - ggruschow

I've got monthly 10-30gb lumps of data I need to upload to S3 for parallel analysis on EC2. It takes days to upload it on my connection, and my ISP will get mad eventually.<p>I ordered a second line to at least avoid pissing off my primary ISP, but I'd rather it was a lot faster. I don't think I can't get yipes, cogent, FiOS, or any of that good stuff where I am (60614), and it's not worth it for me to get a T3 (yet). It seems like I should just be able to drop off a USB drive somewhere that's got bandwidth to spare (any time of day.. I don't care), or maybe there's some multi-megabit upload technology I'm not aware of, or maybe somewhere to do a cheap mid-tower server colo with easy physical access somewhere in Chicago?<p>P.S. The data is already heavily compressed.
P.P.S. I want to do it on the up and up.. I'm not going to sneak it onto someone else's connection.
======
timf
I had a box downtown in the loop (in the Chicago Board of Trade building) here
about 4 years ago:

<http://www.fdcservers.net/ContactUs>

Like all colo, it's not cheap (if you buy for a year, it's $82 a month). But
you get a great pipe and 24hr access (it's manned at all times) to drop DVDs
in the server (you can colo towers, it's not just limited to rackable).

If you only need to do this once or twice a month, I'll bet you could email
them and they'd work something out where you could pay something small to come
in with a laptop or even maybe drop of a whole USB drive for them to upload
(you'd be willing to entrust your S3 keys like that?).

It wouldn't really take that long with their network speeds. Let's say you
could get 10MB/s to S3 from there, 30GB would take, what, like an hour?

Anyhow, they're reasonable, won't hurt to ask.

------
timf
Some similar talk here

[http://solutions.amazonwebservices.com/connect/message.jspa?...](http://solutions.amazonwebservices.com/connect/message.jspa?messageID=75333)

------
pskomoroch
I talked to jeff barr @amazon a while back about these kind of issues (if you
have large datasets to shuffle on/off EC2, is there an efficient & cost
effective way to do it?) - shoot him an email or comment on the aws blog so
they know more people have this need. Amazon public datasets can help with
this for some use cases, for example certain bioinformatics or other academic
use cases.

------
koraybalci
can you just send the data on dvd's with regular mail (ups etc)? of course I
don't know where the server stays, if it's human accessible..

~~~
Herring
to the amazon people? Good luck, I don't think it's possible, but it's the
only option I can think of.

~~~
stcredzero
I smell a business opportunity.

