On: Scaling NSQ to 750B Messages (2016)

[caust1c]

I'm working at Segment now as a Software Engineer in Infrastructure. We just went through a big effort to remove NSQ (in favor of Kafka). Happy to answer any questions.

[coulix → caust1c]

What pushed you to move to Kafka?

[caust1c → coulix]

Reliability, scalability and tooling (in that order).

Basically Kafka provides what NSQ is lacking: https://nsq.io/overview/features_and_guarantees.html

Messages are durable (across nodes, not just persisted to the filesystem). Messages are delivered in-order (for a partition). Consumers know all partitions of a topic from startup, it's not eventually consistent.

NSQ has been great honestly, it was just designed with different use cases in mind.

[nik736 → caust1c]

Was language a deciding factor? NSQ being Go while Kafka is Java.

[caust1c → nik736]

Not at all. We mostly use Go internally for backend services, so if anything that was a point for NSQ.

[zeroc8 → caust1c]

What about NATS Streaming? Is it a viable alternative (now at version 2.0) to Kafka?