Ask HN: Good pub/sub setup for personal projects?

[rschachte]

I’m messing around with a single node k8s homelab I’m running for fun side projects and micro service development.

I’ve done a lot of work with Kafka, but it’s pretty heavy and I really just want a decent pub/sub queue setup between services that would be somewhat reliable.

Is there anything you’d recommend? Open to thoughts on just using Postgres as a queue as well.

[boloust]

At one end of the spectrum: ZeroMQ[1]. No broker to run, and everything can stay within your local network. Just add the library to your microservices and expose a port. It gives you a socket interface with well-engineered queuing primitives. It's enough for most side projects in its simplest form, while also giving you the flexibility to implement a huge number of distributed queuing patterns.

At the other end of the spectrum: just use a managed queue from your preferred big cloud provider. All are reliable enough for this use case, and the cost for most projects would be zero or negligible.

[1]: https://zeromq.org/

[rektide]

Not entirely a real suggestion, but if you want really ultra-fun & easy, patchbay is the most awesome, simplest, easiest-to-use & most versatile/multi-purpose http service for relaying things around/queuing that I know.

Atm it's all in-memory, afaik. There are go & node.js implementations, which I hope some-day we add persistence to.

I'm still working on k8s Persistent Volumes, but in terms of bigger/better software, I hope soon-ish to have a Pulsar Operator[2] going. Kafka is very very exacting & specific, inflexible, but Pulsar's architecture seems much more general & flexible, capable of things like great geo-distribution by virtue of it's underlying Bookkeeper storage layer. It'd be my production go-to for any pubsub details.

[1] https://github.com/patchbay-pub https://patchbay.pub/ https://news.ycombinator.com/item?id=21639066 (362 points, 3 years ago, 79 comments)

[2] https://docs.streamnative.io/operators/pulsar-operator/pulsa...

[remram]

Redis is extremely easy (single binary), though that might not be super compelling if you can run containers. RabbitMQ is good too and has additional features like automatic requeueing if the client stops before acknowledging (and more complex routing rules etc). Both have clients in many languages.

[fendy3002]

uhm, rabbitMQ?

the problem is just Kafka being heavy?

[hallway_monitor → fendy3002]

Probably this. Rabbit is super easy to set up. Super lightweight. You can give it a try in half an hour.

[metadat → fendy3002]

Or a directory of text files, or a relational DB like postgres...

The options are many. It doesn't really matter, just do whatever will meet your needs!

[fendy3002 → metadat]

while possible, those are hard to develop to be a reliable pub/sub messaging though.

[metadat → fendy3002]

It's all hard to make reliable, there are tradeoffs and dark spots around every corner.

When it's a personal project, I suggest settling for "good enough".

[pramsey]

I had a good time with the pq module for a pgsql/python [1] queue. There's a certain brutal simplicity to just having one persistent process, the database, to manage.

[1] https://pypi.org/project/pq/

[winrid]

Why do you want to use a queue between services for a personal project? Just use RPC or similar?

Unless you want to play with queues... otherwise I would use the database, and add redis or rabbitmq later.

[wdb]

Maybe Redpanda or Redis pub/sub?

[joshxyz]

redis streams are cool