
NSQ A realtime distributed messaging platform - harporoeder
https://nsq.io/
======
synchrone
I searched all over the web for a reliable and scalable plain message queue
that could work in a full disk-backed mode, not requiring the data to fit in
memory.

Was very impressed with how little RAM and CPU it consumed under load.

~~~
NicoJuicy
Did you check out NATS?

~~~
synchrone
Was using it in production before. NATS itself is great and reliable.

However disk persistency is done with nats-streaminh and i had big issues with
it's Raft file based store, where it would stop accepting messages after a
short network issue, and not get unstuck unless I destroy the whole nats-
streaming data storage amd start from scratch.

Also raft means 3x disk usage for one queue, since its fully replicated

------
jitl
What is this used for? Is this kinda sorta like a Kafka? Do you implement
message bus with this? Queue task workers? Change data capture stuff like
secondary indexes or materialized views?

As an outsider without deep experience in this space, I struggle to evaluate
this kind of software. When would I want this VS things-that-look-like-it?
What about hosted offerings like SQS? (Maybe I should read a book)

~~~
mnutt
It's good for simple message passing between services, but has some caveats at
scale. One nice thing is that you can read through the source in an afternoon
and get a decent understanding of how it works.

Segment has a good blog post about its pros:
[https://segment.com/blog/scaling-nsq/](https://segment.com/blog/scaling-nsq/)
and later, some of the issues they faced at scale that caused them to move off
of it: [https://segment.com/blog/the-10m-engineering-
problem/](https://segment.com/blog/the-10m-engineering-problem/) My company
has used nsq in a very similar way with a similar number of messages over
almost a decade at this point, and have come to the same conclusions that
Segment did.

------
02020202
i prefer kvdb-backed grpc for messaging to these pubsub projects because there
is always some caveat. either it has at-least-once delivery or at-most-once
delivery and other times it will disconnect clients if they are too slow or
producers are too fast. there is always something. so tailor-made solution
works out the best for me. easy and fast to write. works as one wants it to
work. and with go, it's a piece of cake and also part of single binary, no
external dependencies.

------
est
nsq is a solid piece of technology. 20k conections on a single nsqd, no
complaints.

