Show HN: S3-Accelerator

[dangoodmanUT]

https://github.com/danthegoodman1/s3-accelerator

s3-accelerator is an s3-compatible nvme cache that serves hits in under a millisecond, replicates hot objects, and stays warm when the cluster resizes.

the hot object replication was inspired by groupcache [1], but that has some pretty nasty issues when the cluster resizes

s3-accelerator fixes that by: 1) using rendezvous hashing to reduce the remapping when the cluster loses a node, and 2) when new nodes are added, asking the old parent for a give key for the object rather than going to s3

It also uses learnings from cloudflare that it's better to cache on a second hit than a first hit[2]

It also uses ktls, splice, and sendfile so byte transfer of object chunks never hit userspace. That means it can hit millions of iops, or saturate 200Gb/s network cards, while sipping on CPU

it also is parquet-aware, and has an optional SQS integration to keep the cache fresh if an object is overwritten.

S3 stays the source of truth, and you can write through the cache (it wont cache the write by default)

these features make s3-accelerator uniquely performant, fresh, and s3 costs extremely low.

It's one of those "I'll build it one day" project ideas that Opus 5.5 finally made viable.

[1] https://github.com/golang/groupcache [2] https://blog.cloudflare.com/why-we-started-putting-unpopular...

[munch-o-man]

cool man. I only took a quick look but one question I had was if this runs as a service - since you say to point clients at it - then would it not make more sense to cache on the node the client is talking to s3-accelerator on since that is the first hop? Genuinely curious. I've been working on my own s3/blob project for the past year but it's purpose is a bit different than yours but does involve caching for reads and I'm just curious if you had a reason for that decision that I hadn't considered and might change how I view my own project https://github.com/afreidah/s3-orchestrator (we are both so creative at naming lol). Threw a star on yours so I remember to go back and look at it in more detail later.