This is a great question and something I've been thinking about a lot. Big picture:
Each queue node can operate (mostly) independently, and this is good. As a consumer, I don't really care where my next message comes from, so I can minimize the amount of data that needs to have a "leader".
The only data that needs to be synced is the list of queues, which doesn't change often. If one server is full, it should be able to route a request to another server.
When we downscale, we can use S3/Dynamo (GCS/firestore) to store items and redistribute.
There's more nitty gritty here (what about FIFO queues? What about replication?) but the fact that the main actions, "enqueue" and "dequeue", don't require lots of coordination makes this easier to reason about comapred to a full RDBMS.
This would increase the complexity, but you could always run something like rqlite [0], even if only for the items that require distribution and synchronization.
Or if you truly only need to store simple values in a distributed fashion, you could probably use etcd for that part.
Have you considered libsql? It is a distributed SQLite database, written mostly in Rust. There is a go database/sql and GORM driver available - https://github.com/tursodatabase/libsql
I did not know about libsql! Thanks for sharing. How do you think I should reason about making this distributed - should I use one of the (various) sqlite replicatoin libraries? Or should this be something I roll on my own (on top of some other protocol, such as raft, or something else.)
> How do you think I should reason about making this distributed - should I use one of the (various) sqlite replicatoin libraries? Or should this be something I roll on my own (on top of some other protocol, such as raft, or something else.)
It really depends on the semantics of the MQ you want to provide. There is rqlite if you want a distributed SQLite over Raft.
The question is what sort of guarantees you'd like to provide and how much of latency / performance you are willing to compromise
Each queue node can operate (mostly) independently, and this is good. As a consumer, I don't really care where my next message comes from, so I can minimize the amount of data that needs to have a "leader".
The only data that needs to be synced is the list of queues, which doesn't change often. If one server is full, it should be able to route a request to another server.
When we downscale, we can use S3/Dynamo (GCS/firestore) to store items and redistribute.
There's more nitty gritty here (what about FIFO queues? What about replication?) but the fact that the main actions, "enqueue" and "dequeue", don't require lots of coordination makes this easier to reason about comapred to a full RDBMS.