

Removing your files from S3 can cost thousands of dollars - petewarden
http://one.valeski.org/2010/05/take-2-amazon-s3-file-deletion-fail.html

======
cperciva
This guy must have an incredibly large number of files if 2 XL instances
running deletes for a month only cuts his S3 usage by 10%. I routinely see 700
DELETEs per second from a small EC2 instance; if performance scales linearly
with CPU speed, he should be able to do 700 * 8 * 2 = 11200 DELETEs per
second, or 29 billion DELETEs per month; if that's 10% of his objects, he must
have 290 billion objects stored.

Except that, oops, S3 only passed 100 billion objects a couple of months ago,
and at its current rate of growth is probably still under 150 billion, never
mind 200 or 290 billion.

My guess is that the "FAIL" here is whatever process he's using for deleting
files -- not in S3 itself.

~~~
brown9-2
He claims to need to delete "billions of objects":
[http://one.valeski.org/2010/03/amazon-s3-file-deletion-
fail....](http://one.valeski.org/2010/03/amazon-s3-file-deletion-fail.html)

Would it be naive of me to assume that no single person/company is making up a
few percentage points of all S3 objects?

~~~
jacquesm
I checked to see how many files we have on our main CDN (not S3 hosted, but
just to get an idea), about 500 million files, there are 6 of those boxes so
that's 3 billion files.

We keep three copies of each, but since S3 does that for you I think we
shouldn't multiply by 3.

3 billion files is a lot of files, but it isn't that extreme, he could very
well be telling the truth.

------
danielrhodes
I have also had problems deleting large amounts of data from S3.

The problem that underlies the problem the author was having is that S3 is
quite slow and you can't do batch deletes. On top of that, they limit the
number of API requests you can make per second. In some case it has been
faster to download an object and re-upload it again than do a copy.

~~~
MichaelApproved
That seems to make the case for not needing such a large machine to do the
delete since you're being rate limited. Was your delete CPU intensive? What
setup did you use?

------
ck2
$1500 would buy a really nice server or two on a colo rack.

