
Pulsar – an open-source distributed pub-sub messaging platform - dragonsh
https://github.com/apache/pulsar
======
gas9S9zw3P9c
I've used Kafka a lot in the past and I've been looking at Pulsar. The broker
being stateless is great. I've had issues with moving partitions around in
Kafka.

That being said, I'm getting tired of managing these clunky memory-hungry JVM-
based systems that rely on external dependencies like Apache Zookeeper and
BookKeeper. They may be necessary if you're Yahoo, but I would argue that for
the vast majority of companies, these complex systems create so many
administration headaches that the net productivity impact is negative. I've
spend days or weeks debugging Kafka issues. I almost feel like "big clunky
heavy JVM enterprise software" has become synonymous with Apache projects.

If you don't absolutely need the messaging guarantees I would recommend
looking into NATS ([https://nats.io/](https://nats.io/)) - it's a brokered
systems that's significantly more lightweight and super easy to deploy and
play around with. You can get persistence and delivery guarantees with NATS
streaming, but that part is optional and a bit more early stage.

~~~
KptMarchewa
>That being said, I'm getting tired of managing these clunky memory-hungry
JVM-based systems that rely on external dependencies like Apache Zookeeper and
BookKeeper.

One of my pet peeves with Kafka were CLI tools written in JVM languages. Slow
JVM startup time was killing my focus.

~~~
gas9S9zw3P9c
Yep. I'm using containers and Kubernetes in production so I'm trying to stay
far away from anything JVM-based. On top of being slow to start, memory
hungry, and plagued by GC issues, JVM docker images are always huge. I don't
want two layers of virtualization. If Docker and k8s become more mainstream
that's probably bad news for a lot of JVM projects. Of course, Pulsar itself
is quite old (2013?) so it was built well before any of the containerization
stuff took off.

~~~
zumachase
I’m not really sure what containers/k8s has to do with JVM? Containers are
just name spacing, not virtualization. If you’re running in the cloud, either
way you slice it you’ll have two virtual machines: hypervisor and JVM.

I’d argue that being able to herd your JVM procs like cattle makes them good
candidates for k8s because you can always just set resource limits so they get
purged when the heap becomes too large.

------
fralix
Indeed, Apache Pulsar is a very good project, but as far as I am concerned, I
prefer RabbitMQ in Production env and for little projects, Redis is enough to
take this role.

~~~
nostrebored
Pulsar and Rabbit MQ are fundamentally different products; viewing pulsar like
a push based message queue is missing a lot of the features that make it so
suitable for event driven architectures.

~~~
eternalban
RabbitMQ simply does not like holding on to data. Performance craters with
long queues. RabbitMQ is accurately a “message broker” and quite an excellent
one at that. Tooling is fantastic, and the software (built on Erlang/OTP) is
very reliable.

But it is not a “distributed log”. Pulsar (as Kafka) is built on a distributed
ledger/log. People confuse ‘semantics’ of messaging with “message broker” so
equate various products supporting ‘messaging semantics’.

To your point, and to doubters, simply try building an event sourcing system
(complete with replays to recover) on RabbitMQ and see how that works out!

~~~
jackvanlightly
Not "currently" a distributed log, but we're working on adding that (in a
Rabbity way). We're not trying to compete with the likes of Pulsar or Kafka
though, we're just trying to round out RabbitMQ's functionality to ensure it
remains the best swiss army knife of messaging - and log semantics (and
performance) is now a dominant paradigm.

~~~
eternalban
You're working on Rabbit team now? Interesting! Ironically enough, you
probably have a lot to do with Pulsar gaining mindshare. Thanks for your blog;
very informative.

------
sciurus
Recent discussion of Pulsar vs Kafka:
[https://news.ycombinator.com/item?id=23787958](https://news.ycombinator.com/item?id=23787958)

------
hestefisk
Anyone using this? Looks interesting.

~~~
samdung
Not yet, but reading the docs. Has 'once only delivery' which is good. Still
trying to find if it does FIFO (i think its hard to achieve this in a
distributed system).

~~~
jackvanlightly
Yes it does FIFO.

Just like RabbitMQ, Apache Kafka and many other distributed systems, writes go
through an elected leader, who is able to ensure ordering guarantees.

Specifically with Apache Pulsar, each topic has an owner broker (leader) who
accepts writes and serves readers.

It should be noted that Apache Pulsar supports shared subscriptions which
allow two or more consumers to compete for the same messages, like having two
consumers on a RabbitMQ queue. Here FIFO order cannot be guaranteed for all
kinds of reasons.

~~~
SuddsMcDuff
See also: competing consumers pattern
[https://www.enterpriseintegrationpatterns.com/patterns/messa...](https://www.enterpriseintegrationpatterns.com/patterns/messaging/CompetingConsumers.html)

------
tannhaeuser
What's up with github?

~~~
donut
[https://www.githubstatus.com/](https://www.githubstatus.com/)

> Update - We have identified the source of elevated errors and are working on
> recovery.

> Jul 13, 05:53 UTC

~~~
bArray
A recent version can be accessed here:
[https://web.archive.org/web/20200713060005/https://github.co...](https://web.archive.org/web/20200713060005/https://github.com/apache/pulsar)

------
KoftaBob
Any other Publix fans here that thought this was referring to their sub
sandwiches for a second?

~~~
taywrobel
Given the technical audience of hacker news, I can all but assure you that no
one else here who reads “pub-sub” defaults to thinking “Publix Sub Sandwiches”

~~~
ryaan_anthony
Publix makes great subs but i definitely wasn't thinking about them at first.

