If you are a solo Dev and really want/ need data streams than you can only chose between Kafka and Redis Streams at the moment. While there are a lot of client libraries for Kafka for nearly every programming language, you really don't wanna manage your own Kafka instance and, to be fair, it really feels like overkill.
So RabbitMQ seems to fit into the same spot as the Redis Streams do:
Easy manageable and usable streams, even for smaller projects. Really hope that this will be usable in many different client libraries soon.
There are some missing features compared to Kafka (no dynamic rebalancing of partitions being the top one), but they're also rolling out some new features that Kafka does not have, such as transformation pipelines .
NATS's brand new JetStream  feature is also looking very promising. It uses Raft internally. NATS itself is rock solid, and JetStream builds on that foundation.
NATS also has very interesting support replication topologies, meaning you can build graphs of streams that each feed in other streams, so you can do the publishing and consumption in different locations, with different availability constraints.
I took a look at redpanda this week, it sounded nice on paper, but them being a young company, I am concerned about what "gotchas" we are going to run into.
We did push it to more than 10k partitions, but I honestly don't remember how that affected it; that's when I discovered that partitions cannot be dynamically rebalanced, which meant we'd have to change the way we would use it.
Interestingly, we also did a similar test with NATS JetStream, which did start struggling around 10,000 consumers. (A consumer in NATS is similar to a partition, as it has its own Raft group.) What I tried to do with JetStream goes against the grain a bit, mind you; I still think it's an excellent piece of software.
It's riding the wave of CNCF cloud-buzzword projects but don't let that scare you off -- generally that means that it is actually really easy to set up and operate, and does most of the things you'd expect via well structured pre-inserted configuration which is a plus. The devil is still in the details though, so read the docs thoroughly to make sure it fits your usecase.
For regular messaging
There's documentation on this bug/feature and it looks like using NATS streaming is a way to fix this. I will admit that it's a bit annoying to figure out the difference between NATS streaming, jetstream and Liftbridge but I don't think this issue affects all 3. Jetstream is essentially NATS streaming built into the NATS binary itself, so switching to it should produce the feature set you're looking for.
JetStream is impressive and looks very promising. I did some stress testing recently and found some performance issues and possible bugs, but I wouldn't hesitate to put it into production.
RabbitMQ plays in the field of traditional general purpose messaging system. NATS was specializing on performance. Kafka focused on streams. That is a focus and includes trade offs. Which is all fine. That is all I want to address.
So much of my life would be better right know if you didn't just describe me.
Then you probably have Kafka connect running somewhere. That's another handful of servers. Maybe Kafka streams is a few more servers. Then you're going to have servers that collect events and publish them to the server. How many more servers is that per environment?
Congratulations, you now have a state of the art event streaming enterprise grade platform, and 20 servers to manage. Better hope your company gets on board with the real time model, otherwise you're now the owner of these pet servers until the end of time.
What's that, you ran into a rare kafka bug that caused your offsets to be lost after a reboot and now the bus is pushing millions of messages down the pipe to all the consumers? Wow, that sure sucks, hope you can juggle your day job with this massive production issue.
I use google pubsub as a stream - it's just me developing our tools. I find it very easy to use and just works.
But maybe that doesn't count as a 'stream' under that definition?
I use GAE to accept and pass to pubsub super fast 1000s/second bursty webhook data, pass to pubsub, which is triggers cloud function to write to DBs. Cloud function retries if there is a write error or timeout or something.
It worked so well I've now just used this as a kind of micro-service for all DB writes I have to do. Now also parsing out other 'processing' services that don't need to respond with data to the request, like for instance an example 'service' we verify and format cell phones with twilio and then update that user profile.
Best of both worlds :)
- You can have a team working on the core product sending messages and another working on integrations triggering actions.
- If your integrations fails, your core product is not impacted. You can also replay old messages once you've fixed the stream consumer.
- Having streams allows you to do all kind of experiments too, you can connect a new project to a stream and go through a week of data almost instantly and see the result... rince and repeat as much as necessary
So, I'm taking bets how many days it takes until some teams decide that any further development is impossible without this feature. Yes, it's been a long day.
I for one on the ops side will be pushing to upgrade ASAP because the sooner devs adopt this the sooner I can tear down Kafka and have one less thing to maintain.
> Streams in RabbitMQ are persisted and replicated. This translates to data safety and availability (in case of the loss of a node)
Let's talk about the elephant in the room: Do streams replicate the same way queues do? If you ever lost a node while you had a few GBs of data in queues you will know that bringing a node back will sync all data over the net while completely blocking ANY operation on the queue until this process finishes. Please don't recommend quorum queues, they lack important features as well.
W.r.t quorum queue features set we are working on Message TTLs. Priorities we'll have to see. We want to provide something there but it may not be a priority queue as provided by classic queues as this isn't the best way to do priority based messaging.
Getting insights into the roadmap for this would certainly be interesting...
I don't know about RabbitMQ, but with Apache AMQ there is message grouping, which is kind of similar, but not quite the same. With Kafka it's unavoidable, which is good.
If you had unbounded storage you could perhaps.
From what I can still recall, AMQP (RabbitMQ) looked great to me until Redis Streams came along. RabbitMQ always seemed more heavyweight in various ways, whereas Redis Streams was very easy to pickup and get going with.
Redis Streams isn’t simple, but I always found RabbitMQ to be more complex and to have more gotchas.
For example, on the test 3-node cluster (c2-standard-16) it achieved publish rate of almost 5 millions messages per second.
Queues let many writers put messages into a single topic. Then, many readers come to that topic, and pop messages off. Each message goes to just one reader. You could use this for background jobs. For example, if you were running YouTube, you might handle uploads this way: the uploaded videos are put into a queue, and workers process them to transcode them and make them playable on the website. You don't want the uploads to be processed more than once.
Append-only logs let many writers append messages onto a single topic. Then, many readers may replay the entire history of messages whenever they want. Each message may go to many readers, and may even go to the same reader multiple times if they want. You could use this to build a "message bus" where you want lots of things to happen after an action. For example, every time a user "likes" something on facebook, maybe we want to notify the content-producer, notify their friends, and update some recommendation algorithms - three different things that we want to do each time, and we don't want errors in one to block the others.
To elaborate a bit, crypto is backed by tokens specifically to ensure integrity in a distributed network (on machines you don't own)
If you own the DB you don't need this.
You can use an open source version of AOL (Append-Only Log) SW for free, alternatively you can use a managed cloud version, but there you will have to pay with something called USD which is a social construct 1:1 pegged to a crypto-tokens like Tether USDT, MakerDAO DAI, and other so called "stablecoins" ;)
it's absurdly reliable, it can scale super well because erlang is great (as long as you just give it enough hardware).
the only issue i had was clustering -- back in the day, it was a pain to cluster rabbitmq. nowadays it's not that hard and works really well!
I switched to Sensu's experimental Redis queue config and never had that issue again. Ended up running 3x small Sensu servers in each region, each running Sensu + Redis in HA mode. Bulletproof, if properly configured.
Maybe clustering has improved, but having to use it as part of Sensu cluster put me off it.