
Show HN: Lightbus – Python/Redis Message Bus for Eventing and RPCs - adamcharnock
https://lightbus.org/
======
adamcharnock
Hi Hacker News. I started developing this over two years ago when I become
dissatisfied with what I felt was a lack of good options for an easy-to-use
Python messaging system. I didn’t think it would take this long (of course!),
but I’ve certainly learnt a lot about messaging and distributed systems. I
tracked my progress in issue number 1 on GitHub, and I feel pretty happy that
I now have 98 of 98 tasks complete! [1]

I’ve been using this in production now for about a year and it has faired very
well. I found a few weird bugs along the way, but it has felt stable for quite
a while now. I’m therefore finally making an official release. I wanted to
call this version 1.0, but I’m going to play it safe and call it 0.9 for a few
months and see if anything crops up in other peoples’ environments.

Before embarking on this I spent a long time talking to other developers to
see if anything similar existed. The most similar I could see was Nameko, but
I didn’t feel like that ticked my boxes. Lahja is another one which I saw pop
up more recently. I’ve also written a short comparison with Celery, as it
something I’m often asked about [2].

I’ve also spent a lot of time writing up the docs. A crude word count
indicates I’ve written 23,400 words of narrative documentation :o

I was also keen to not strictly limit this too Python. I’ve therefore
documented the Redis interactions required for other languages to interface
with Lightbus[3], albeit without the various niceties that Lightbus provides).

Constructive feedback is very welcome!

[1]:
[https://github.com/adamcharnock/lightbus/issues/1](https://github.com/adamcharnock/lightbus/issues/1)

[2]: [https://lightbus.org/explanation/lightbus-vs-
celery/](https://lightbus.org/explanation/lightbus-vs-celery/)

[3]:
[https://lightbus.org/reference/protocols/](https://lightbus.org/reference/protocols/)

------
omginternets
We ([https://bespoke.app](https://bespoke.app)) use Lightbus in some
supporting infrastructure, and despite starting with a less stable version,
we've been quite happy with it.

Firstly, the maintainers are extremely responsive, and have gotten most issues
fixed in less than a day. IMHO, the devs are being particularly conservative
in calling this 0.9 instead of 1.0 -- which is arguably a good thing! Here's a
quick/informal experience report. Happy to go into details.

Pros:

\- It's very "Django"-y. The declarative interface reads and writes like
Django ORM code, so there's very little context switch. It should feel
familiar to Django programmers.

\- It's built on "boring" tech. The default transport layer is Redis. Most of
us don't need a distributed queuing system like NSQ or Kafka. Config is YAML.

\- The API is designed to favor ease-of-use over virtually everything else,
and yet there's actually very little magic under the hood. There's a bit, but
we've found it pretty easy to reason about. Default options are sensible, and
easily tunable.

\- The documentation is good. It's a bit spartan for some of the more complex
things (e.g. writing your own client in $YOUR_FAVORITE_LANGUAGE).

Cons:

\- Our biggest concern has been with logging. Message-bus architectures have
all the difficulties of cooperative multitasking, with the added problem that
events can also go missing. Good logs are invaluable in debugging this kind of
problem. My understanding is that this is on the roadmap.

All in all, I think it's a safe bet for pure Python projects. If you're a
polyglot shop, you _may_ want to consider something else in the short term,
but Lightbus is _very_ capable of solving the problems that message-bus
architectures were designed to solve [0].

[0] e.g.: "how can these isolated web apps share data in a sensible way".

~~~
adamcharnock
Hey there! Thanks for the comment, I'm glad its been going well. I'll reply to
a couple of your points below:

> The documentation is good. It's a bit spartan > for some of the more complex
> things

Yes, I've documented the protocols[1] but I haven't gone into much more
detail. The docs may also need some iterating on as/when people try to develop
new clients.

> Our biggest concern has been with logging. > Message-bus architectures have
> all the > difficulties of cooperative multitasking, > with the added problem
> that events can also > go missing. Good logs are invaluable in > debugging
> this kind of problem. My understanding > is that this is on the roadmap

Yep. Two things are on the roadmap that are relevant to this:

1\. Revamping the logging system, likely to have an option to support
structlog 2\. The ability to see a causal trace for each message (Message C
was caused as a result of Message B, which was caused by Message A)

I think no. 2 should be straightforward, and I think no. 1 is just going to
take a grim sense of determination and deleting a bunch of old logging code.

[1]:
[https://lightbus.org/reference/protocols/](https://lightbus.org/reference/protocols/)

