
Amazon S3 Transfer Acceleration - p-pat-ni
https://aws.amazon.com/blogs/aws/aws-storage-update-amazon-s3-transfer-acceleration-larger-snowballs-in-more-regions/
======
toomuchtodo
TL;DR

Newer, larger "snowball" appliance (current = 50TB and newer 80TB version), as
well as access in additional international regions.

Network optimized S3 uploads (faster inbound transfers) for an extra 4
cents/GB.

Sidenote: I'm curious if you can get the same speedup for inbound transfers by
using a Cloudfront distribution in front of your S3 bucket for no additional
charge (no cost for inbound transfer on either service).

~~~
cmwright
We played with this a bit using both cloudfront and other cdns. While we did
see some small improvements, it wasn't nearly as good as we were expecting on
the upload side. It may be fixed now, but there were also many issues getting
the proper headers to pass through cloudfront to s3 for chunked uploads.

~~~
samstave
This might not be the right question, but:

Did you test using S4CMD with many threads as opposed to S3CMD?

------
disbelief
Anyone know if the quoted cost of $0.04/GB for Transfer Acceleration is
instead of the $0.03/GB for standard S3 ingress, or in addition to it? Ie. is
the final cost per GB $0.07 or $0.04 when using Transfer Acceleration?

 _Edit:_ as pointed out to me below, there actually is no ingress cost for
standard S3 uploading. The $0.03/GB is monthly storage. So the cost of this
service is $0.04/GB instead of $0.00.

~~~
koolba
> Anyone know if the quoted cost of $0.04/GB for Transfer Acceleration is
> instead of the $0.03/GB for standard S3 ingress, or in addition to it? Ie.
> is the final cost per GB $0.07 or $0.04 when using Transfer Acceleration?

S3 ingress is free. You're thinking of storage pricing.

~~~
disbelief
Ah yes of course, thanks. I took too brief a glance at the S3 pricing page.

------
s9ix
Did anyone else have some pretty terrible results from the speed tester? I was
pretty optimistic seeing their sample screenshot - am I missing something?
[http://i.imgur.com/UwNBGf3.png](http://i.imgur.com/UwNBGf3.png)

~~~
aroch
If it makes you feel better, the accelerated ones are slower for me!

[http://i.imgur.com/n6DhyHj.png](http://i.imgur.com/n6DhyHj.png)

~~~
CobrastanJorji
Same here, got 11%-24% slower in most regions, and only 3% faster in US-
EAST-1. Sao Paulo got 60% faster, though.

Regardless of my results, I've gotta hand it to Amazon on a really kickass
speed comparison page. That's a great user experience. It immediately tells me
whether I should be using this or not from my particular corporate network
without requiring me to waste a day or so trying it out.

------
johnp_
Secure Connection Failed

An error occurred during a connection to aws.amazon.com. Peer attempted old
style (potentially vulnerable) handshake. Error code:
SSL_ERROR_UNSAFE_NEGOTIATION

Really,Amazon?

[https://www.ssllabs.com/ssltest/analyze.html?d=aws.amazon.co...](https://www.ssllabs.com/ssltest/analyze.html?d=aws.amazon.com)

No PFS, TLS 1.0 only, No HSTS or HPKP ??

------
Benjamin_Dobell
> The name of the bucket used for Transfer Acceleration must be DNS-compliant
> and must not contain periods (".").

Source: [http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-
acce...](http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-
acceleration.html#transfer-acceleration-getting-started)

> The bucket names must match the names of the website that you are hosting.
> For example, to host your example.com website on Amazon S3, you would create
> a bucket named example.com.

Source: [http://docs.aws.amazon.com/AmazonS3/latest/dev/website-
hosti...](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-
custom-domain-walkthrough.html)

...

Looks like they don't want you replacing CloudFront with Amazon S3 Transfer
Acceleration - which is a royal pain for some of my use cases (internal facing
websites that don't need a full CDN).

~~~
re
S3 Website Hosting is distinct from CloudFront; you can create a CloudFront
distribution backed by a bucket with any name. You can combine the two, but it
isn't required.

[http://docs.aws.amazon.com/AmazonCloudFront/latest/Developer...](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/MigrateS3ToCloudFront.html)

[http://docs.aws.amazon.com/gettingstarted/latest/swh/website...](http://docs.aws.amazon.com/gettingstarted/latest/swh/website-
hosting-intro.html)

------
sudhirj
Looks like they're using the Cloudfront infra to have files sent to s3 via
cloudfront. Uploads to a cloudfront edge down the street will be a lot faster
than the s3 datacenter.

Will this allow the HTTP sliding windows to be optimized more in any way? Any
idea if this could approach UDT upload speeds for large files?

------
strick
Still no ipv6 for S3?

~~~
ec109685
Do you have IPv6 only devices you'd like to support?

------
Cozumel
I went to try it but don't see it as an option!

~~~
vaibs
It will be released by tomorrow.

~~~
Cozumel
Thanks!

------
narfz
is there a way to run the benchmark from CLI? i would like to run the
benchmark on our servers to see if we can get an improvement.

~~~
ec109685
You can setup a socks proxy pretty easily with ssh -D 1080 <your server name>,
then set localhost:1080 as your socks proxy and run the test.

~~~
oasisbob
That's a great solution if you want to benchmark your proxy.

------
dantiberian
Do I understand this correctly that Cloudfront is still the better option for
fronting static assets like JS and CSS?

~~~
Benjamin_Dobell
You don't actually have a choice:

[https://news.ycombinator.com/item?id=11531357](https://news.ycombinator.com/item?id=11531357)

~~~
mryan
That is not correct. You can host a website from an S3 bucket without using
CloudFront. Although of course, downloads will come directly from the S3
bucket and not from a CF edge location.

~~~
Benjamin_Dobell
Huh, just what did you think I was saying? Of course you can put CloudFront in
front of S3 website hosting!!!

You _can 't_ use S3 Transfer Acceleration with S3 Static Website Hosting.

~~~
mryan
My apologies, I misread your comment.

------
ghshephard
Does anybody have a sense of how fast uploading to S3 can get? I'm on a
gigabit link here in Singapore, uploading to Singapore instance of S3 (via
ARQ) - and I'm disappointed that I rarely see better than about 40
Mbits/second.

~~~
philliphaydon
I'm in Singapore, uploading to Oregon, 100mb, took minutes , I changed to
upload via signed cloud front requests, takes 15s to upload 100mb now...

~~~
ghshephard
I presume you mean 100 MB (or just plain 100 MBytes) - so, 53 megabits/second.
So a little better, but nowhere near 1 gigabit (which I presume you have, it's
so cheap here - not that it seems to be useful for anything other than great
speedtest results)

~~~
philliphaydon
I use homeplugs because the router and computer are in different rooms. So I
only get about ~300mbit from my room. I've never actually tried from next to
the router.

My point was just that I get better results using CloudFront than using S3 or
Multi Upload S3.

Sharing what I can :)

------
nbevans
I wish Azure would improve Blob Storage. It is really a shadow of S3's
capabilities.

------
nxzero
What's the fastest & cheapest way to get data off Amazon's cloud services?

~~~
morgante
It depends entirely on the quantity of data.

For huge datasets, Snowball will be fastest & cheapest (literally shipping
data).

~~~
vvanders
As they used to say, never underestimate the bandwidth of a van filled with
harddrives.

~~~
cortesoft
Never underestimate the bandwidth of a station wagon full of tapes hurtling
down the highway.

~~~
nxzero
Always IP over Avian Carriers:
[https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers](https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers)

&&

Sneakernets

[https://en.m.wikipedia.org/wiki/Sneakernet](https://en.m.wikipedia.org/wiki/Sneakernet)

------
itaifrenkel
Is the improvement also applicable when uploading files from the s3 management
console into an accelerated bucket?

------
sandGorgon
I'm not seeing this in my buckets. is this going to roll out gradually ?

~~~
tylermac1
It's turned on tomorrow.

------
peteretep
Still no uploading of gzip'd data to S3, amirite?

~~~
mryan
What stops you from uploading gzip'd data to S3?

~~~
peteretep
It stays gzip'd once you've uploaded it.

~~~
mryan
Well to be fair, that is not the same as "you can not upload gzip'd content".

Your complaint seems to be "S3 doesn't automatically gunzip gzip files that I
upload", which sounds like the desired behaviour to me. i.e. if I ever upload
gzipped content to S3, it is because I want it to be served compressed over
HTTP, or because I am moving a compressed backup file to S3. In neither case
would gunzipping be desirable, although I do appreciate that compressing
during the transfer, and decompressing on the other end, could save some bytes
during transfers.

~~~
peteretep
My complaint is that I can't upload a gzip'd file and have it decompressed on
their end, yes.

Yes, the other things I could have meant would have been odd, so I'm not sure
why you spent time typing them out.

~~~
mryan
> Yes, the other things I could have meant would have been odd, so I'm not
> sure why you spent time typing them out.

It seemed clear that you were either confused about what features were
supported by S3, or not expressing your complaint correctly. The reason behind
typing them out was trying to politely point this out, while describing some
of the real-world use cases for using gzip'd files in S3 (partially to
highlight why it would not make sense for S3 to gunzip files automatically).

I felt this would be a more useful comment than "you are wrong, you can upload
gzip'd files to S3", or just downvoting your comment for being incorrect.

------
nxzero
"What's a Snowball?"

Animated YouTube by Amazon explains:
[https://youtube.com/watch?v=9uc2DSZ1wL8](https://youtube.com/watch?v=9uc2DSZ1wL8)

~~~
shortstuffsushi
Even with that, I don't get it.

"Moving lots of data either requires a huge pipe, or a ton of storage disks."

With that, they offer their Snowball device, which, if I'm understanding
correctly, holds up to 50TB (now 80TB), which they physically ship to you, and
then you ship back to them. How does this fix either of the constraints (disk
space / connection pipe)?

~~~
brianwawok
So you have 50TB of images to send me and a 10mbit internet pipe. Is it faster
to FTP me the file or mail it on a hard drive?

~~~
shortstuffsushi
To you and @nxzero, I'm not debating whether it's better or worse, I'm trying
to find out if I'm actually understanding it right. From some of the other
comments, it appears that it is just a matter of them sending you a physical
disk -- it makes sense now.

~~~
brianwawok
Yes they mail you a fancy disk, you fill and mail back.

------
negrash
Amazon S3 Transfer Acceleration is natively supported in FREE CloudBerry
Explorer [http://www.cloudberrylab.com/blog/amazon-s3-transfer-
acceler...](http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceleration-
support-in-cloudberry-explorer-v4-6/) and CloudBerry Backup
[http://www.cloudberrylab.com/blog/amazon-s3-transfer-
acceler...](http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceleration-
in-cloudberry-backup-4-8-2/) from day one. Check it out!

