Indeed this is not an easy problem. And our s3-compatible system do support the atomic rename with extended protocol in a graceful way, see the demo with our tool [1].
Thanks for the analysis. Interestingly when we first released our low latency s3-compatible storage (1M IOPS, p99 ~5ms)[1], a lot of people asking the same questions why we tried to bring file system semantics (atomic object/folder rename) to s3. We also got some feedback from people who really need FS sematics, and added POSIX FS support then.
aws S3FS is using normal FUSE interface, which would be super heavy due to inherent overhead of copying data back and forth between user space and kernel space, that is the initial concern when we tried to add the POSIX support for the original object storage design. Fortunately, we have found and open-sourced a perfect solution [2]: using FUSE_OVER_IO_URING + FUSE_PASSTHROUGH, we can maintain the same high-performance archtecture design of our original object storage. We'd like to come out a new blog post explain more details and reveal our performance numbers if anyone is interested with this.
A lot of good insights here. I am also wandering if they can just simply put different jobs (unclaimed, in-progress, deleted/done) into different directory/prefix, and rely on atomic object rename primitive [1][2][3] to solve the problem more gracefully (group commit can still be used if needed).
well I checked the code in their repo，but could not find any snippets as mentioned in the post，which made the claimed perf numbers very suspicious. There was also no cluster configration details in their claim，but you can verify our performance numbers by simply following the deploy steps in the readme.
FractalBits is an S3-compatible object storage system designed for high performance and low latency. Using our custom-built fractal ART metadata engine, it delivers up to 1 million 4K read IOPS for single bucket with p99 latency ~5ms, at significantly lower cost than AWS S3 Express One Zone. Unlike standard S3, FractalBits provides native atomic rename support for both objects and directories. We are happy to open source our API server implementation (which support both actix and axum framework) today and welcome any feedback!
[1] https://github.com/fractalbits-labs/fractalbits-main/tree/ma...
reply