
Building a Relational Database Using Kafka - rayokota
https://yokota.blog/2019/09/23/building-a-relational-database-using-kafka/
======
humbleMouse
This is fetishizing complexity for no reason. Makes for an interesting blog
post, but if any company told me they were doing this I would run the other
direction immediately.

~~~
thinkersilver
Great article. I think you might be taking a subtle point in the article for
granted. Namely, the ability to assemble a fairly complex distributed system
from open source components.

Democratisation of complex machinery like the raft and consensus algorithms,
schedulers, append logs, query optimisers and so on is a superb thing. It is
something that has only been possible in the last 3/4 years and would have
been devilishly difficult before then without significant upfront work.

Fetishising complexity might be a bit strong here. Seeing more posts like this
is a good thing.

~~~
mLuby
You make a good counterpoint: it's awesome that this can be done concisely
enough these days to fit in a _hobby_ blog post. That's new and demonstrates
the approachability of these tools.

At the same time, OP is right: a _business_ doing this is making poor
engineering decisions.

~~~
kjeetgill
We don't know that yet! The post explains what they did but not the why of the
problem. Maybe they have a good reason, maybe they don't.

But no need to jump to conclusions. I've seen plenty of bs fragile ducktape
databases. Some of them exist because because someone wanted to hand-roll
everything. And some exist because they were afraid to invent anything and
ended up hacking together a bunch of garbage MySQL nodes. Because they didn't
want "reinvent the wheel".

Specifics matter!

------
outworlder
I am amazed by the number of negative comments this article is getting.. on
_Hacker_ News of all places.

I don't see the article claiming people should drop PostgreSQL and use this,
it's just asking people to give it a try if they are interested.

Much more interesting would be to discuss what capabilities would be different
if this were a mature product.

For instance, this sounds interesting:

> One advantage of using Kafka is that multiple servers can all “tail” the
> same set of topics. This allows multiple KarelDB servers to run as a
> cluster, with no single-point of failure. In this case, one of the servers
> will be elected as the leader while the others will be followers (or
> replicas). When a follower receives a JDBC request, it will use the Avatica
> JDBC driver to forward the JDBC request to the leader. If the leader fails,
> one of the followers will be elected as a new leader.

------
lichtenberger
I like the idea, to use Apache Kafka as the underlying log. It's in principle
what Martin Kleppmann suggests.

As I also want to use a distributed log in the future: Do you know Apache
Pulsar or the underlying BookKeeper, which I think was especially made for a
distributed WAL?

I might either want to use Apache BookKeeper or Apache Pulsar for a
distributed log for scaling my Open Source temporal database, too. Furthermore
I'd like to expose the API for streaming changes into the Browser or wherever
you want to :-)

~~~
eternalban
I've been itching for a while to build something on top of BookKeeper's
DistributedLog. I suggest ditching the Pulsar and just using the DL.

~~~
lichtenberger
I've already written with one of the core committers to BookKeeper and yes, I
think I'll use it.

That said, I'm always looking for users and contributors to
[https://sirix.io/](https://sirix.io/) or
[https://github.com/sirixdb/sirix](https://github.com/sirixdb/sirix). That
would be super awesome, but I'm sure you have your own ideas already for using
BookKeeper.

I'm currently not sure if I first want to build a frontend (I'm a backend
engineer, but would like to learn some TypeScript along with using Vue.js and
D3js), to interact with SirixDB and to build interactive visualizations to
compare revisions of JSON- or XML-resources in SirixDB (stored in a binary
format of course highly optimized for space-efficient snapshots).

I think as I'm lacking users it might be more useful, but for sure I'm at
least as eager to put forth the idea of scalable SirixDB databases :-)

So, I'd like to use BookKeeper most probably (single writer, read your own
write consistency, using synchronous -- for a quorum -- and asynchronous
replication for the rest, exactly once semantics...). The thing I don't like
is that we also need ZooKeeper, but yeah.

BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)

~~~
eternalban
This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I
highly recommend it:

[https://jack-vanlightly.com/blog/2018/10/2/understanding-
how...](https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-
pulsar-works)

> The thing I don't like is that we also need BookKeeper, but yeah.

Word on the street is now you can swap the consensus module in ZooKeeper.
[update: just re-checked Pulsar's dev channel. Apparently one can use etcd
instead of zookeeper.]

> sirixdb

Sirix sounds interesting but is a bit close to home to a (currently dormant)
temporal DB project of my own. But the real issue is I usually do significant
OSS contribution if I'm on the bench and not working.

Wish you success in your effort!

~~~
atombender
Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably
easier to set up and manage, in my experience; not needing to manage another
JVM component is also a big win. Thanks for sharing!

~~~
eternalban
To be clear, it is BookKeeper that can alternatively use etcd not Pulsar.

[https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-
sto...](https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-store/)

[https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...](https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed70f6feb02ef14d2d032abb9e)

~~~
atombender
Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK
gives it an edge over Kafka.

I have to admit that I'm not entirely clear on what Pulsar adds, given that
BookKeeper implements the distributed log. Pulsar looks like it's a slightly
higher-level abstraction that gives you a more traditional topic-based pub/sub
mssaging API on top of BK, plus things like schemas and "functions"?

From what I can tell, BK supports log streaming just like Kafka, so you can
use it to build a pub/sub system if you don't need all of Pulsar's features —
right?

~~~
eternalban
Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd),
and Pulsar also has its own (low traffic) consensus via ZK. The article cited
above does a very good job of addressing the advantages of Pulsar's
architecture over Kafka's. Pulsar also provides for user defined functions
(that can run on the broker). This is reminiscent of Postgres's User Defined
Functions.

BK provides ledgers, DistributedLog provides log semantics on top of those
ledgers. Pulsar builds a full blown messaging/streaming system on top of that.

Per this thread's OP, it seems to make more sense to use BK/DL directly if you
want a WAL component to build a DB.

~~~
atombender
Thanks. Right, I know Pulsar is built on top of BK — it's just unclear how
much extra value it adds.

I'm looking at using BK for a WAL-like component for a document store. It does
involve multiple (many!) consumers tailing the log, but unlike your typical
Kafka setup, there's need for no partitioning.

I was unsure of whether BK provides an API that's nice to use for non-Java
languages like Go and Rust, which is what we use. Apache projects written in
Java (or Scala), particularly those in the Hadoop ecosystem, tend to be very
JVM-centric, and often they're impossible to use from other languages because
there's no non-Java API. Kafka is an outlier here, but it certainly started
out with a Java-only client and it took years for non-Java clients to mature
enough to be usable.

I thought I saw that BK had a REST API, but now I see that it's just an admin
API. From what I can tell, it's exactly like every other Apache Java project —
there's a big monolithic Java client that's tightly coupled with the server,
and to write a client in any other language you'd have to reverse-engineer it
and essentially port the whole thing, because it's a "fat client" — it looks
like it talks directly to ZooKeeper and "bookies", and has a lot of logic
built in. The Bookie protocol looks like it's binary and hand-coded (not gRPC
or anything nice). So disappointing! /rant

Edit: Looks like the files I was looking at are generated from Protobuf. So,
yay. Unfortunately, that's just the Bookie protocol. The main client is still
fat and in Java.

~~~
lichtenberger
Regarding the question if Apache Pulsar or Apache BookKeeper for me it's not
so clear which of both to use. I think it adds a lot of value if you add the
WAL to the public API. Forgot how it's called, I think change data types
(CDTs) without searching on google!? So basically you have the stream of
events and expose these for use by applications :-)

------
tener
Interesting article, but I think it is missing a solid comparison of such
"overlay SQL" with more traditional database engine.

It isn't enough to show something can be done. One must also think if it
_should_ be done.

~~~
swagonomixxx
The author is doing this purely for fun (I think, I'm not the author). They
link to this:
[https://www.confluent.io/product/ksql/](https://www.confluent.io/product/ksql/)
which is apparently a production tested system that provides a SQL interface
to a Kafka backend.

~~~
dominotw
> provides a SQL interface to a Kafka backend.

Not sure if thats an accurate description of ksql.

------
xmpir
Why would I want a relational database living in Kafka? MySQL or Postgres are
great at what they do.

~~~
xjoins
I came here to ask this question. It's a really solid technical article. But
apart from the sake of doing it, why would anyone ever want to? I'm not
suggesting there's no scenario in which one would want to build a SQL database
using Kafka, I'm just asking if anybody knows of one.

~~~
wefarrell
One use case for Kafka is log aggregation. Using SQL to analyze logs would be
one use case.

------
pram
From personal experience, considering the reliability of Kafka, I’d implement
this in reverse. Most relational databases are far more resilient in
operation.

One idea I’ve had in the past is to implement a Kafka and Zookeeper layer for
FoundationDB.

~~~
EdwardDiego
Interesting, what were the failure modes you encountered?

