
Show HN: Liftbridge – Lightweight, fault-tolerant message streams - tylertreat
https://liftbridge.io
======
berkayozturk
For those who are wondering the difference between Kafka/Pulsar and Liftbridge
(from the docs):

"Liftbridge was designed to bridge the gap between sophisticated but complex
log-based messaging systems like Apache Kafka and Apache Pulsar and simpler,
cloud-native solutions. There is no ZooKeeper or other unwieldy dependencies,
no JVM, no complicated API or configuration, and client libraries are just
gRPC. More importantly, Liftbridge aims to extend NATS with a durable, at-
least-once delivery mechanism that upholds the NATS tenets of simplicity,
performance, and scalability. Unlike NATS Streaming, it uses the core NATS
protocol with optional extensions. This means it can be added to an existing
NATS deployment to provide message durability with no code changes. The
ultimate goal of Liftbridge is to provide a message-streaming solution with a
focus on simplicity and usability."

~~~
ignoramous
NATS' author, u/tylertreat, wrote a series of articles on _how to build a
distributed log_ that are pretty insightful:

Part 1, Storage Mechanics:
[https://news.ycombinator.com/item?id=15983185](https://news.ycombinator.com/item?id=15983185)

Part 2, Data Replication:
[https://news.ycombinator.com/item?id=16021876](https://news.ycombinator.com/item?id=16021876)

Part 3, Message Delivery:
[https://news.ycombinator.com/item?id=16101223](https://news.ycombinator.com/item?id=16101223)

Part 4, Trade-offs and Lessons:
[https://news.ycombinator.com/item?id=16181963](https://news.ycombinator.com/item?id=16181963)

Part 5, Sketching a New System:
[https://news.ycombinator.com/item?id=16215788](https://news.ycombinator.com/item?id=16215788)

~~~
atombender
I think you mean Liftbridge's author. NATS was created by Derek Collison [1],
co-founder of CloudFoundry.

[1]
[https://news.ycombinator.com/user?id=derekcollison](https://news.ycombinator.com/user?id=derekcollison)

------
kiwicopple
This is very cool OP. Amazing work.

For anyone looking for something _similar_ , we are adding streams to Postgres
- [https://github.com/supabase/realtime](https://github.com/supabase/realtime)

It isn’t an exact comparison but it has some of the features you see here too
- you can use wildcards listen to: all changes in your database, schema-level,
table-level changes, or row-level. The server is built with Elixir, so you can
have thousands of listeners connected via web sockets.

It’s not as lightweight as OP’s solution but PG is very familiar.

~~~
davedx
This is really exciting, nice! We've just started using postgres for our new
services and the other day I was just thinking to myself "wouldn't it be great
if I could listen to certain database tables because I can then build a real-
time data warehouse really easily". How far along is the implementation?

~~~
kiwicopple
Thanks for the feedback @davedx. Actually it's being used in production by a
few companies already (not small companies either). We have a few things we
want to do before it's "enterprise ready":

    
    
        - set up elixir clustering (i.e. autoscaling)
        - run benchmarks
        - fine-grained client auth
        - ability to listen to multiple databases
    

> then build a real-time data warehouse really easily

This is the nice thing about PG. It's an operational database that can handle
the load of an analytics database. Easy choice! If you need a hand trying this
out, message me: copple [at] supabase.io

------
tylertreat
Author here. I started Liftbridge after working as a core committer on NATS
and NATS Streaming (as well as my experience using Kafka in the past). In
particular, implementing replication in NATS Streaming and drawing from that
experience/lessons learned. Happy to answer any questions!

~~~
polskibus
Could you compare Liftbridge to Akka with persistence module and or streams?
I'm looking for a lightweight, embeddable Kafkasque thing (esp interested in
exactly once delivery) that I can use to glue various services together.

~~~
kitd
Not OP but you may find MQTT useful.

[http://mqtt.org/](http://mqtt.org/)

~~~
polskibus
Thanks, is there an option to make guaranteed delivery with MQTT?

~~~
kitd
What you want is the quality of service "QoS" flag set at connection time
(edit: sorry, correction, publish time):

[https://docs.oasis-
open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.h...](https://docs.oasis-
open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901103)

Bear in mind that there are always limits in how guaranteed a "guaranteed
delivery" actually is. "Exactly once" means that the broker has received it
and the client has acknowledged that. What happens beyond the broker is
outside the scope of the protocol and would have to be handled at the
application level, eg using transaction ids, etc.

------
kodablah
I've been watching this with some interest as the successor of NATS streaming.
One question I have is how does it fair in potential-high-latency,
geographically distributed situations? E.g. what if I had dozens of servers at
the edge worldwide wanting to receive events from one another?

~~~
paddybyers
We've been experimenting with Liftbridge at a range of scales and workloads,
and clusters that are distributed across a single region perform very well.
However, it's not really suitable for operating a cluster distributed across
multiple regions because the latency adversely impacts the bandwidth of the
Raft protocol that's used for cluster-wide coordination; this impacts the rate
at which streams can be created, and the rate at which changes to In-Sync
Replica sets (ISRs) can be propagated. As suggested, a federation of clusters,
each covering a single region, would be required.

------
bamazizi
The project has basically jumped from v0.0.1 to v1.0.0 within 4-5 months ...
but how does it fare with reliability, throughput, ordering and deliveries?

Is it PROD ready? are there any use-cases or project sizes this utility is NOT
recommended for? i.e. sensitive events such as payment or ecomm related vs.
web traffic noise logs

------
fcanesin
How does it compare in performance?

