
Amazon S3 - Object Size Limit Now 5 TB - jeffbarr
http://aws.typepad.com/aws/2010/12/amazon-s3-object-size-limit.html
======
fredoliveira
Thanks for bringing the discussion over to HN, Jeff.

While I don't have a use for the new object size limits (the data we store at
S3 is on average of only a few Kb per object - even though we store millions
of them), I can't wait to see what people (mainly the scientific community) do
with this. Bumping the object limit to 5TB means huge datasets can now be
stored, processed straight from and added back to S3 - removing size limits is
the final piece of the puzzle to "trivialize" big data processing.

PS: It's been fun, and exciting, and unexpected to follow Amazon AWS' growth
in the past few years. I remember back when AWS was first announced thinking
of how odd it was for a company like Amazon to open its infrastructure like
you guys were doing. A little while later, we were fortunate enough to be
among the very first companies to use EC2 (Goplan was the first 3rd party
product to run on EC2 back in the day - it got us on Wired Magazine). These
days, AWS has become an integral part of most startups I see and products I
work with. That is a huge deal.

------
gruseom
How do you access a file that large? Presumably you don't have to stream it
from S3 to EC2 in its entirety in order to do computations with it. Does that
mean you can have random access into it in order to work with a chunk at a
time?

~~~
btn
S3's REST API supports the HTTP 'Range' header to pluck bits of an object out
(with a similar facility in its SOAP API).

------
bobf
Although I wouldn't argue there is an immediate need for 5TB objects, raising
the previous 5GB limit is undoubtedly useful.

~~~
mashmac2
I definitely see a connection between this and their recent addition of
Cluster GPU instances... big rendering files, scientific data sets, etc...
[http://aws.amazon.com/about-aws/whats-
new/2010/11/15/announc...](http://aws.amazon.com/about-aws/whats-
new/2010/11/15/announcing-cluster-gpu-instances-for-amazon-ec2/)

------
cperciva
Hmm... 5 TB is enough to store 12 trillion digits of Pi.

~~~
jeffbarr
So what are you waiting for?

~~~
cperciva
I'm busy playing with other things on EC2 right now. :-)

