
Moving 25TB data from one S3 bucket to another took 7 engineers and 2 full days - simonpure
https://www.reddit.com/r/aws/comments/irkshm/moving_25tb_data_from_one_s3_bucket_to_another/
======
darthShadow
RClone. Probably the best tool I have come across for interacting with cloud
storage solutions.

Move Docs:
[https://rclone.org/commands/rclone_move/](https://rclone.org/commands/rclone_move/)

S3 Docs: [https://rclone.org/s3/#amazon-s3](https://rclone.org/s3/#amazon-s3)

Supports parallel server-side copies and deletes (no server-side moves,
unfortunately) so this would have been much faster.

~~~
rsync
Are there any cloud storage providers that have 'rclone' built into their
environment ?

So I could, for instance:

    
    
      ssh user@rsync.net rclone s3:/some/bucket gdrive:/blah
    

or:

    
    
      ssh user@rsync.net rclone s3:/source/files /my/rsync.net/account
    

Well I'll be damned! It appears that there _is such a provider_ :

[https://www.rsync.net/products/rclone.html](https://www.rsync.net/products/rclone.html)

[https://www.rsync.net/products/universal.html](https://www.rsync.net/products/universal.html)

~~~
brylie
> NO Charges for ingress/egress

Is this too good to be true?!

Thank you for sharing! Yet another gem found in HN comments.

~~~
throwaway9d0291
No, the tradeoff is that even with volume, rsync.net storage costs are
$15/TBmonth [0], which is ~3x managed alternatives [1,2] and ~10x DIY
alternatives [3]. To be fair, ZFS snapshots are "free" but whether that's
worth 3x cost will depend on your data.

The lack of bandwidth charges sounds really nice but [1,3] have that too and
unlike [1,3], Rsync.net doesn't support protocols that are suited for
bandwidth-intensive applications (like serving large files to end-users). It
only supports SSH-based transfers [4].

[0]: [https://www.rsync.net/pricing.html](https://www.rsync.net/pricing.html)

[1]: [https://wasabi.com/cloud-storage-pricing/](https://wasabi.com/cloud-
storage-pricing/)

[2]: [https://www.backblaze.com/b2/cloud-storage-
pricing.html](https://www.backblaze.com/b2/cloud-storage-pricing.html)

[3]: [https://www.hetzner.de/dedicated-rootserver/matrix-
sx?countr...](https://www.hetzner.de/dedicated-rootserver/matrix-
sx?country=us)

[4]:
[https://www.rsync.net/products/platform.html](https://www.rsync.net/products/platform.html)

~~~
rsync
It is true that our service is more expensive.

If you look at the pricing on that rclone page[1] you'll see that at quantity
it is as low as 0.5 Cents Per GB / Month.

As for bandwidth intensive protocols, we have HPN-SSH[2] patches built into
our environment which allow for high bandwidth transfers, over SSH, over long
WAN links.

[1]
[https://www.rsync.net/products/rclone.html](https://www.rsync.net/products/rclone.html)

[2] [https://www.psc.edu/hpn-ssh](https://www.psc.edu/hpn-ssh)

~~~
throwaway9d0291
> If you look at the pricing on that rclone page[1] you'll see that at
> quantity it is as low as 0.5 Cents Per GB / Month.

That's good but at the volume needed to reach it, I imagine that you're in the
realm of "contact us and get a special deal" at other providers (e.g. AWS,
GCP, Azure) too.

Plus, we haven't even got to the "you have to pay for capacity, not usage"
aspect of rsync.net pricing.

Pricing is the big thing I wish you'd fix. Your service has a bunch of cool
features but they're just not worth that price, not when I can get most of the
same features from Hetzner for a fraction of the cost.

> As for bandwidth intensive protocols, we have HPN-SSH[2] patches built into
> our environment which allow for high bandwidth transfers, over SSH, over
> long WAN links.

That's cool (honestly, not being condescending) but my point was that for most
of the reasons you'd be excited about free bandwidth, rsync.net isn't a viable
solution. For example you can't host images, videos, software updates, machine
learning datasets or other large binaries and serve them over HTTP on the
public internet.

~~~
rsync
"rsync.net isn't a viable solution. For example you can't host images, videos,
software updates, machine learning datasets or other large binaries and serve
them over HTTP on the public internet."

Correct. We don't do these things and we never will.

When you 'nmap' an rsync.net storage array, you get:

22 TCP

... _and that 's it_. There are no other services running, or offered. There
are no interpreters in the environment. The filesystems are mounted
noexec,nosuid.

We do one very simple thing _and that 's it_.

~~~
throwaway9d0291
And were you to do that simple thing at a competitive price, it'd be great.

But you don't.

To be competitive you either need to drop prices or do more stuff to be worth
the money.

~~~
mekster
To be fair, storage cost is their only pricing, meaning lowering it will have
a great impact on their income when others charge for bandwidth and extra
support.

But having no other cost than the storage itself is relieving and having the
service running for nearly 20 years gives you a peace of mind on their
reliability. I also hear their support is good

I would want a bit more modern web interface though. BorgBase does look
interesting to me, except only 2 years of operation can't tell about its
reliability and longevity.

------
jiggawatts
Simply copying lots of files over an extended period can break your
applications elsewhere:

 _" Amazon S3 automatically scales in response to sustained request rates
above these guidelines, or sustained request rates concurrent with LIST
requests. While Amazon S3 is internally optimizing for the new request rate,
you might receive HTTP 503 request responses temporarily until the
optimization is complete. This might occur with increases in request per
second rates, or when you first enable S3 RTC. During these periods, your
replication latency might increase. The S3 RTC service level agreement (SLA)
doesn’t apply to time periods when Amazon S3 performance guidelines on
requests per second are exceeded."_

That's a fun gotcha...

------
don-code
I used to work for a company which used S3 as its data backend - copying lots
of files in and out of S3 is how the product worked. It was also how we
enabled QA/STG and the CI/CD pipelines to interact with some of that data -
sanitizing it was just a matter of removing or rewriting metadata.

We never solved this problem well.

Our first go was a naive `aws s3 cp --recursive` between the source and
destination buckets - this executes all files to copy sequentially. Not good
for performance when you have hundreds of thousands of files.

Our next go was to use some custom Ruby scripts to do it, executed under JRuby
so that we could take advantage of Java threads to multithread the process.
This also didn't work well - the Ruby SDK's `CopyObject` implementation seems
to hang some threads if you transfer multiple files concurrently. Our team was
primarily Ruby-based, so going pure Java was an option that we didn't pursue.

We then heard about S3 Batch Operations, and launched an unsuccessful POC,
because Batch Operations don't work on files larger than 5GB. That's an
immediate non-starter for our use case.

We ended up with a horrible hack after many person-weeks of work on this
problem: 1) Get a list of all of the files in the source bucket, and put them
on a queue. 2a) If a file is small, copy it using a native Java/Ruby threaded
worker. 2b) If the files is large, fork an `aws` CLI process and have it `s3
cp ...` the file.

I'm not surprised _at all_ that copying data took these seven engineers two
full days.

~~~
rapsey
So you clearly picked the wrong tool for the job and spent an obscene amount
of effort hacking around your tool instead of picking a better one.

~~~
jml7c5
What alternative would you have proposed?

~~~
lmm
Buck up and learn enough Java to write this basic program. It's really not a
particularly complex or hard language.

~~~
ed25519FUUU
(Sarcasm) You should “buck up” and learn Rust or C. With a real language you
won’t have to deal with the overhead of a runtime and will really be able to
saturate the resources.

~~~
lmm
I can and have written Rust, C, and indeed Ruby where warranted. Surely any
self-respecting programmer would do the same?

------
kondro
I’m surprised none of the engineers either knew about, or thought to ask AWS
support, S3 Batch Operations or S3 Replication.

Both of which could’ve completed this process in a relatively short amount of
time in a completely hands-off way.

When I Google for “move large S3 bucket” the first result I get is an AWS
article which goes through all the best options:
[https://aws.amazon.com/premiumsupport/knowledge-
center/s3-la...](https://aws.amazon.com/premiumsupport/knowledge-
center/s3-large-transfer-between-buckets/)

~~~
booleanbetrayal
If you read the thread, they mention that they had a requirement to empty the
original bucket within the maintenance window, as well.

~~~
altdatathrow
Their biggest issue was letting someone who didn't know a thing about S3
dictate the rules. Revoke all access to bucket ~= everything is deleted.

Then they could set a lifecycle rule, wait a day, and it would be gone for
real.

~~~
dmlittle
From my experience if you have a bucket with billions of objects it will still
take a few days after the objects are deleted by the lifecycle rules to be
able to delete the bucket. The objects are non-recoverable but I believe due
to the eventual consistency nature of S3 you can't delete the bucket until the
objects are all fully deleted.

~~~
altdatathrow
Yeah, S3 is frankly a disaster. People use it like a filesystem and if you
have billions of objects, it very quickly becomes a nightmare to manage. Size
of data is a complete non-issue, it's all about the number of objects.

~~~
grogenaut
I will point out deleting that much data from harddrives isn't a fast
operation either if you were storing it on drive. Same for tape.

Sure you can destructively destroy the drives. But you could also just use KMS
on the S3 bucket object and then toss the key and close the buckets. Gonezo.

~~~
the8472
> I will point out deleting that much data from harddrives isn't a fast
> operation either

Dropping a btrfs subvolume or snapshot is pretty fast. ditto for zfs.

~~~
danielheath
That doesn't help if you want the data to be gone instead of hidden.

~~~
Dylan16807
If it's encrypted then destroying the key gives you a blank-with-noise drive.
There's no equivalent for S3.

------
manigandham
GCP and Azure both have much better built-in tooling that would make this a
few clicks. Their storage system design is also much better. It's unfortunate
that the industry has standardized around S3 just because it's a first mover
rather than pushing Amazon's product to get better.

------
ed25519FUUU
Fun problem. Without replication, here's how I would do it.

A pipeline: S3 list worker(s) (see below for optiizations) -> SQS -> aws
lambda (s3 sync) -> SQS -> aws lambda (s3 delete of original key).

Now the secret: to make it really go vrrooom you need to sufficiently
parallelize the list operations. For this enable S3 storage analytics on the
bucket to get a manifest of all of the files a few days beforehand. Use that
list to partition the prefix/key space evenly into 20 or 30 (or 100, or 200)
workers each with start and end key prefix.

With SQS, you can make sure the sync and delete actions succeed, and with
lambda they'll run in a serverless fashion as fast as you can feed the queues.

Amazon will absolutely throttle you when making a lot of requests. Reach out
to them ahead of time. Make your lambda code robust, so that a failure is
idempotent lean into the retries.

------
jlarocco
Something sounds fishy about this. What exactly were the 7 engineers doing
during this time? And none of them thought to Google it or call support?

TBH, I won't claim I know the best way to do this, but I'm 100% sure a call to
Amazon support would sort things out. Copying data between buckets is common
enough I'm sure they have a good way to handle this.

~~~
hinkley
I’m sure you’ve run into a less egregious version of this at some point yeah?
Automation doesn’t always take longer than doing the task. But it does mean
zero progress for part of it.

In any group of 7 engineers there is at least one who is more than happy to do
a repetitive task. The pay is the same. Whether there is one who would rather
do anything but that? I think that depends on luck, and whether the other 6
chased them off already.

The “it’s not so bad” crowd always fits in easily, but they might not be that
effective when push comes to shove.

------
perrygeo
Even if you were moving 25TB of data on a server under your physical control,
it could be a significant engineering effort. Maybe not 14 person-days but at
least a few.

What's amazing is that some teams assume the same operation will be more
performant in the cloud, despite being wrapped in layers of HTTP-based API
abstractions.

~~~
brundolf
If it's within the _same_ cloud, though, why wouldn't there by an internal
shortcut so it doesn't have to go fully "out" then "back in"?

Heck, why does it have to physically move at all? Why doesn't this effectively
come down to a rename within AWS' system, like when you "move" a file within
the same hard drive?

~~~
SteveNuts
That's probably possible, but not publicly exposed. Like a lot of the people
on the Reddit post said, reach out to support. They have a LOT of data and
knobs to turn, in my experience.

~~~
dmlittle
Not OP but we also had to move a relatively large S3 bucket (significantly
larger than 25TB) and unfortunately AWS doesn't have a way to change the
ownership of the S3 bucket. My guess is it has to do with how the underlying
system stores the objects.

We ended up writing a Go program to copy the objects from one bucket to
another to help with the parallelization of migration. This was also before
AWS had announced S3 Batch operations so I'm not sure how much better it would
be to use that today. The deletion of the bucket also took us over a week.
Even though we had deleted all objects in the bucket due to the eventual
consistency nature of S3 we weren't allowed to delete the bucket until all
objects were fully removed from S3. All we could get from AWS support was to
wait a few more days and reach back if we couldn't delete it then.

Edit: depending on your object naming scheme you might also run into the S3
prefix rate limits.

------
booleanbetrayal
TL;DR - The delete is the hard part. You can use S3 batch operations with an
inventory list to do the copy very quickly. Alternatively, you can setup
replication and "touch" each file using a self-copy CLI command, once the
replication policy is in place. For the deletion, you're sort of stuck with
lifecycle policies, which would take a day or so to clear out the old bucket,
but could be supplemented with manual interaction, I'd imagine. There probably
needs to be a better mechanic for completely wiping buckets. Last I checked
there was not.

~~~
grogenaut
encrypt all objects. Wipe bucket? just toss key and delete bucket, done.

Use generated bucket names not fancy bucket names, buckets are cows not pets.

~~~
booleanbetrayal
Only, that's not actually deletion, nor would it have helped their business
requirement:

> It is a 3rd party application that puts data into that origin bucket. They
> needed the bucket to be empty before the new version gets activated. And
> they wouldn't use another bucket. Something out of our control

------
awiesenhofer
I had a similar task once, tried it with awscli and s3cmd first, then took
about 5 minutes to google for a multithreaded alternative and found s4cmd[1],
problem solved. (tmux to keep the session running)

Makes me kinda doubt at least parts of this story - not one of these 7
engineers thought about googling?

Edit to add: Agree with the basic premise though - s3 can be real slow when
when you actually need to work with your data.

[1] [https://github.com/bloomreach/s4cmd](https://github.com/bloomreach/s4cmd)

~~~
gfody
or even faster [https://github.com/peak/s5cmd](https://github.com/peak/s5cmd)

~~~
miniman1337
s5cmd is super fast

------
orf
Lots of weird comments here. I’ve done this solo with a 200tb bucket in about
a day. A rando “aws s3 cp” obviously isn’t going to cut it.

They had the weird requirement where they also needed to empty the bucket,
which makes things slightly more complex. You’d create a small 5 line lambda
that would copy an object to the new bucket and then delete it. You’d then
invoke this with a batch operation.

If you can relax the empty requirement, then you’d just use the batch
operation to copy without needing a lambda and set a lifecycle policy to
delete the objects. You’d need to do a bit of manual work to copy-delete
objects created since the last inventory ran, but that is fairly simple.

You can use a tool like rclone but that’s still going to be quite slow
compared to a batch operation, especially if you have a higher number of small
files.

Alternatively, you would set up replication between the buckets and just
handle the deletion within the window. S3 can delete 1,000 objects per call,
meaning emptying a bucket is pretty fast.

Or, lastly, you’d use “s3 cp” with an appropriate number of threads on a
machine within AWS.

------
hn2017
I had a similar problem, except I had to download about 600GB (mainly from
small <30kb files) from a client's s3 to on-prem. It took a painfully long-
time using the AWS-CLI.

Never really found any good solutions but fortunately it wasn't urgent so it
wasn't a big deal. But this should be easier.

------
ars
I often copy (not synchronize) remote files using rsync instead of scp because
rsync will batch the small files, and scp does not.

Or zip up (or tar) the small files, copy over the large archive, then unzip.

They are having the same issue here: A lot of small files each transferred one
at a time, instead of in a streaming batch.

~~~
awiesenhofer
You are getting downvoted because tfa is about the s3 protocol.

All of these commands unfortunately wont help you with s3. If you want to
manipulate that from the command line you are basically stuck with aws cli,
s3/s4cmd or scripting it yourself using a library.

~~~
ars
I wasn't giving them advice, I was just talking about a similar problem in
another context.

------
levischoen
Do you hear the sound of clinking chains? That’s vendor lock in ;-)

------
jedberg
That's because they did it in the worst possible way.

AWS has built in tools to do this in the background where they take care of
everything for you.

------
dekhn
I moved a ton of data in and out of S3 when I worked at $STARTUP. You just
need to tune the s3 options a bit. Haven't had any problems using a beefy
machine, often saturating 1Gbps with small files, or 10Gbps with large files,
using a single beefy machine.

------
nikolay
We are all hostages of the clouds!

------
exabrial
Yikes. I miss Unix and rsync

------
billman
I think you can use replication to accomplish this.

------
crb002
They have Snowball for this, up to 50 TB. [https://aws.amazon.com/getting-
started/projects/migrate-peta...](https://aws.amazon.com/getting-
started/projects/migrate-petabyte-scale-data/services-costs/)

Also, throwing a swarm of lambdas to chunk the files into about 250mb
temporary zip files would help.

~~~
kyawzazaw
Isn’t snowball for migrating on prem data to AWS? Not AWS S3 to another S3
with the listed reqs

