This is accurate! A lot of people have spent a lot of time trying to build a good file system abstraction on cheap, S3 storage. However, Regatta differs from these solutions in two important ways. First, Regatta is a shared, durable caching layer that sits between your instances and S3. This means that Regatta is able to efficiently perform operations (like directory renames) and provide strong consistency to other file system clients (whereas s3fs and other FUSE file systems would need to actually perform those operations in S3 for other clients to see the output). Secondly, Regatta is designed to support all file system operations. This means that you can do file locking, random writes, appends, and renames -- even when they aren't efficient to perform on S3.
Super interesting product. I have a couple of questions:
In terms of storing in s3 - is that in your buckets? Sound like the plan is to run the caching on your infrastructure, are there plans to allow customers to run those instances themselves?
Presumably the format within s3 is your own bespoke format? What does the migration strategy look like for people looking to move into or out of your infrastructure? They effectively pull everything down from their s3 to the local “filesystem”?
I love this because it allows me to highlight the parts of the system that I'm most excited about. The Regatta caching runs on our infrastructure, but it connects to buckets that our customers control. We read and write data into the customer's bucket in a regular, native (not bespoke) format -- so you can connect a Regatta file system directly to a bucket that already exists, with data in it, and use that data from a file system without any data migration!
