I searched all over the web for a reliable and scalable plain message queue that could work in a full disk-backed mode, not requiring the data to fit in memory.
Was very impressed with how little RAM and CPU it consumed under load.
Was using it in production before. NATS itself is great and reliable.
However disk persistency is done with nats-streaminh and i had big issues with it's Raft file based store, where it would stop accepting messages after a short network issue, and not get unstuck unless I destroy the whole nats-streaming data storage amd start from scratch.
Also raft means 3x disk usage for one queue, since its fully replicated
What is this used for? Is this kinda sorta like a Kafka? Do you implement message bus with this? Queue task workers? Change data capture stuff like secondary indexes or materialized views?
As an outsider without deep experience in this space, I struggle to evaluate this kind of software. When would I want this VS things-that-look-like-it? What about hosted offerings like SQS? (Maybe I should read a book)
It's good for simple message passing between services, but has some caveats at scale. One nice thing is that you can read through the source in an afternoon and get a decent understanding of how it works.
Segment has a good blog post about its pros: https://segment.com/blog/scaling-nsq/ and later, some of the issues they faced at scale that caused them to move off of it: https://segment.com/blog/the-10m-engineering-problem/ My company has used nsq in a very similar way with a similar number of messages over almost a decade at this point, and have come to the same conclusions that Segment did.
We use it for internal messaging between services. Particular between systems owned by different teams. It's reliable, easy to configure, have good out of the box backoff. We have used Kafka in the past and it just felt too heavy for what we needed.
i prefer kvdb-backed grpc for messaging to these pubsub projects because there is always some caveat. either it has at-least-once delivery or at-most-once delivery and other times it will disconnect clients if they are too slow or producers are too fast. there is always something. so tailor-made solution works out the best for me. easy and fast to write. works as one wants it to work. and with go, it's a piece of cake and also part of single binary, no external dependencies.
Was very impressed with how little RAM and CPU it consumed under load.