

How to directly upload files to Amazon S3 from your client side web app - tadruj
http://codeartists.com/post/36892733572/how-to-directly-upload-files-to-amazon-s3-from-your

======
mjtokelly
Direct uploading like this has been possible for at least two years. However,
in our tests it turned out that Amazon was silently losing some files >100MB
(and almost all files ~1GB).

On the AWS forums it was clear that AWS was aware of this consistent problem,
but was not able to fix it, and not willing to document it.

They eventually released "multipart upload", which remains the only reliable
way to get large files to S3. Unfortunately, multipart upload is nearly
impossible to implement as a web app (short of resorting to e.g. Java).

~~~
tomsaffell
>Unfortunately, multipart upload is nearly impossible to implement as a web
app

I am right now writing a library to do just that. I already have a prototype
working, and we'll probably open source our library once it's reasonably
tested. But perhaps you know something I dont... is there anything in
particular that makes it "nearly impossible"?

~~~
mjtokelly
Sounds like my understanding must be out of date as of this year! (Which makes
me very excited about your upcoming library.) My conclusion was based on
research ~1 year ago finding no OSS or commercial software supporting it, and
forum posts for those projects by developers sadly explaining why they just
couldn't do multipart yet.

My bad experience with files disappearing was entirely with web-based POST
requests, ~2 years ago. Large file transfer from EC2 to S3 was reliable, but
our POST requests on slower connections (even those that were very reliable)
would return with a false report of success.

------
brettcvz
Alternatively, use Filepicker.io.

filepicker.store(myFile, {location: "S3"}, function(FPFile){console.log("File
stored in S3 at "+FPFile.key+" and available at "+FPFile.url);});

~~~
philfreo
unfortunately they don't return the full url to the file on YOUR s3 bucket
(but you can piece it together)

~~~
brettcvz
Our concern is that if we return the direct S3 url, people less experienced
will assume that the S3 url should work by default, when it will 403. So we
return Filepicker.io urls which alias to your S3 bucket

~~~
philfreo
then you should have an example IAM Policy in a very easily accessible
location

overall as someone who is currently using filepicker, i would recommend people
avoid it if all you want to do is upload to files to your own bucket without
any transformations like image resizing

~~~
matb33
You could leverage zencoder to generate your thumbnails. Send the job as a
test so it's free (limited to 5 seconds but that's ok since we only want
thumbnails) and specify your thumbnail settings as desired. You could even set
the clip length to one frame so the video encoding step is nearly skipped.
Since zencoder runs on AWS, you should get near-LAN transfers between S3 and
zencoder

------
philfreo
Thanks for posting... I was looking for something just like this:

<https://github.com/tadruj/s3upload-coffee-javascript>

~~~
tlrobinson
I really like CoffeeScript, but do we really need to advertise every
JavaScript library as a CoffeeScript library as well?

Feel free to use CoffeeScript to implement your libraries, but lets just call
it a JavaScript library and assume everyone who uses CoffeeScript will
understand they can use any JavaScript library.

~~~
tadruj
I think we need to. It deserves a credit, because we use it and we didn't pay
for it. CoffeeScript makes our lives easier, almost eliminates bad practices
and removes brace clutter from the screen, making the code more readable,
understandable, better.

When having to choose between a library written in JavaScript and a library
written in CoffeeScript I choose the latter, because it statistically has a
probability to be a better library, just because the coder didn't have a
chance to use bad JavaScript practices, when writing it. Assuming I usually
don't have time to read the code of the libraries I'm using.

------
ayushgta
Thanks for posting this. I was looking at adding uploads just this morning, so
this is very timely. It looks like if you combine the direct upload to S3 as
outlined in this article with bitline (<http://www.blitline.com/>), you get
some pretty cool abilities. I especially like that bitline supports batch
operation (for example create five sizes of my image) in one call.

------
modarts
Tangentially related, but does anyone know of an easy way to push a github
repo to an S3 bucket (configured as a public site)?

I'm pretty new at git/github, but understand that the solution may involve a
post commit hook that somehow calls an S3 file upload service.

~~~
gdubs
I was actually reading up on this earlier; I was led to jgit
[http://stackoverflow.com/questions/7031729/publish-
to-s3-usi...](http://stackoverflow.com/questions/7031729/publish-to-s3-using-
git)

~~~
modarts
Sorry, I should have clarified a little better. I'd like to avoid installing
anything on a server to make this happen.

(My development workflow/environment is completely decoupled from any servers
I control: I use Parse.com as my backend with static HTML pages served from
S3, and cloud9 as my IDE, with Github as my source host)

------
secabeen
Here's the other page on the AWS documentation on this:
[http://s3.amazonaws.com/doc/s3-developer-
guide/RESTAuthentic...](http://s3.amazonaws.com/doc/s3-developer-
guide/RESTAuthentication.html)

------
dgoodlad
I did a demo similar to this a few months ago:

<https://github.com/dgoodlad/s3-photo-upload-demo>

<http://s3-photo-upload-demo.throwawayapp.com>

------
vinhboy
Last week I was trying to figure out how to do this with dropbox. Anyone know
a way?

~~~
tadruj
Dropbox has a filepicker <https://www.dropbox.com/developers/chooser> , you
can also use filepicker.io for more services support. But Dropbox doesn't
support per-request CORS as I read on their Forums.

~~~
vinhboy
G@#$D!T I wished I had known about this last week. I already implemented my
own file browser! Thanks for the tip though.

------
pswenson
Coolest would be an ability to have 1) a great filepicker UI (drag target +
multi file upload w/ progress bar) 2) resizing/thumbnails in the browser 3) no
server side other than s3

guessing #2 isn't possible?

~~~
carson
#2 is possible but #3 isn't unless you have a completely open bucket and that
wouldn't be good.

~~~
brettcvz
I hate to be so self-promoting, but <http://jsfiddle.net/GTYX5/2/> should do
everything you're asking for

~~~
pswenson
filepicker.io is cool. But your backend service wouldn't be needed if the
image uploading & resizing straight to s3 could be done in the browser. Your
product would simply be a javascript lib.

------
pud
Any recommendations for how to show a progress-indicator to the user during a
direct-to-S3 upload?

~~~
carson
I wrote the example referenced at the bottom of the article and it has an
example of indicating progress to the user. You can see it in app.js here:
[https://github.com/carsonmcdonald/direct-
browser-s3-upload-e...](https://github.com/carsonmcdonald/direct-
browser-s3-upload-example)

