
Kafka Spawns Open-Source KarelDB - lelf
https://www.datanami.com/2019/09/30/kafka-spawns-open-source-kareldb/
======
derefr
I've been looking into Kafka lately, because I have a need for "the thing that
Kafka is in practice" (not a message queue per se; rather, an indefinitely-
durable event store that supports message-queuing API semantics for
consumers.)

And, while Kafka _does_ do this, its infrastructure needs are kind of
ridiculous when your producers 1. produce events in clean batches, and 2.
could re-produce those events, in the exact same batches, if prompted.

Ideally, rather than having to basically have a whole second DBMS in the form
of a Kafka cluster (stateful nodes with big HDDs, carefully backed up, needing
repair when any die), what I would love is a Kafka-like "event store with MQ
semantics", which is in turn backed by an _object store_. Basically, like
Datomic's segment-storage architecture; or like WAL-E in its use-case as an
intermediary for Postgres replication. Unlike either of those, though, MQ log-
segments don't need any additional metadata (e.g. checkpoints, indices), so
they can just be chucked straight in object storage, and MQ nodes can base
their view of the world entirely on what happens to be in the object-storage
bucket.

Of course, this kind of system only makes sense if you have no need for
durability of new messages (because e.g. the producer still has the data that
went into them, and can resend them), only old messages. Which is kind of an
unusual requirement, I suppose, so I don't really expect anyone else to go
building this sort of thing for me.

~~~
atombender
Agreed. For me, Kafka's reliance on ZooKeeper is the most annoying one.
ZooKeeper is difficult to manage. And both Kafka and ZK, being JVM apps, are
memory hungry, and the heap is difficult to tune — they need lots of headroom
to avoid accidental GC pauses or OOM events.

Kafka also puts some of the complexity burden on the client. It's been a while
since I wrote anything for it, and I believe you can tell it to manage the log
position for you now. But by default I believe a client needs to talk to both
Kafka and ZooKeeper (or store the log position somewhere else).

There's definitely room for a simpler solution for those who can't afford the
complexity of managing Kafka.

I looked at Apache BookKeeper [1] recently. It has a nice design and now
supports Etcd for consensus. Unfortunately, like Kafka, it relies on a fat
client, and that client is currently written in Java, making BK impossible to
use from a language like Go or Node.js.

Apache Pulsar [2] is a layer on top of BookKeeper that provides pub/sub
semantics and high-level features like functions and schemas, and offers
client support for multiple languages. On the other hand, Pulsar requires both
BookKeeper and ZooKeeper (and doesn't support Etcd). In theory you could run
zetcd to emulate ZK with Etcd, but... it gets a little bit ridiculous at this
point.

I've looked at NATS Streaming, but its clustering story looks a little
unfinished.

I've previously built a system that built a log on top of Postgres. You can
achieve a decent approximation of Kafka using transactions and LISTEN/NOTIFY,
and with table partitioning and replication you can even get some scale out of
it, especially if you're able to distribute independent logs across completely
separate Postgres instances. But it's ultimately not as good as a dedicated
store.

I particularly like your idea that — if I understand it correctly — data can
be stored independently of the log itself.

[1] [https://bookkeeper.apache.org](https://bookkeeper.apache.org)

[2] [https://pulsar.apache.org](https://pulsar.apache.org)

~~~
math
There is currently (much anticipated!) work underway to remove Kafka's
Zookeeper dependency: :
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-
Managed+Metadata+Quorum)

As of v0.9 (about 4 years ago), clients have no use for a direct connection to
Zookeeper, and don't maintain one.

Fat clients are more effort to make, which is a significant downside, but this
is abstracted from the user and Kafka has critical mass - the client story
outside Java is improving dramatically. A fat client allows for optimal
availability, throughput, latency.

~~~
atombender
Thanks, that's promising.

A fat client makes a lot of sense, but if it's written in Java, then only JVM-
based languages benefit. A client written in C, C++ or Rust — or any other
language with a similarly good C-interop story — could easily be consumed by
just about anything except, ironically, Go.

~~~
math
there are pros and cons to having a C dependency. the main benefit is
reliability (get it right once, leverage that across languages). Another
benefit is usually performance. The main practical gotcha is that security
dependencies are incompatible across platforms, and the best you can do is
distribute a number of builds of the C dependency compatible with common
platforms (but that is workable).

the go story isn't as bad as you might think. there is a performance hit with
the interop, which is of the order of 20% iirc - but throughput still massive,
and this is not of practical significance. The Confluent go client (a binding
to librdkafka), is used to push much of the volume through two of the largest
Kafka installations in existence.

there is a PR open on the confluent go client that will include builds of
librdkafka in the github repo, meaning this will 'just work', without the need
to install librdkafka separately in most common scenarios.

note: i work on kafka clients (including minor contributions to the confluent
go client).

~~~
atombender
Thanks -- I thought the Go stack switch overhead was more than 20%, that's not
awful. A lot better than JNI, I think?

------
manigandham
KCache is an in-memory Java cache where all writes are sent to a Kafka topic
and then applied locally by any instances listening to that topic, providing
an eventually-consistent cache. Great for scaling a small dataset that can fit
on a single-node with easy replication and a full log of all updates in the
Kafka topic.

KarelDB adds relational querying, schema/table management, transactions, and a
SQL interface on top of KCache by using various Apache open-source projects
(Calcite, Omid, Avatica, Avro). Here's a blog post from the author that has
more details: [https://yokota.blog/2019/09/23/building-a-relational-
databas...](https://yokota.blog/2019/09/23/building-a-relational-database-
using-kafka/)

A typical database (like Postgres) or cache (like Redis) has a WAL where all
writes are applied and this log is also used for replicas. KCache and KarelDB
are interesting experiments in doing this from scratch using open-source
components, although using Kafka as a durable log of system-agnostic changes
is already very common. In fact, it's probably the most used feature of Kafka
Connect with various database connectors and changefeed streams.

~~~
ako
Sounds like Kareldb would make a good alternative to rocksdb, when combining
tables and streams in Kafka streaming?

------
amdelamar
So, KarelDB makes use of KCache (among other OSS). Which so happens that Kafka
uses. But KarelDB does not use Kafka and vice versa. Did I get that right?

EDIT: After reading the GitHub README, it does appear KarelDB makes use of
Kafka. Its unclear to me if its running kafka brokers underneath, or depends
on an existing cluster.

------
slowmovintarget
[https://yokota.blog/2019/09/23/building-a-relational-
databas...](https://yokota.blog/2019/09/23/building-a-relational-database-
using-kafka/)

------
netfl0
Can anyone explain why this might be exciting? (genuinely curious)

~~~
thanatos_dem
Ever wanted a database with all the operational complexity of Kafka, and more?
Well now you can have it!

It’s exciting as someone more in the SRE side, because you’d need to pay be an
obscene amount to support this.

~~~
diehunde
Isn't that "side" just a bunch of clicks on a wizard on some cloud provider?

------
mcepl
Will the next version be called Josef? (to continue line of the famous authors
from Prague)

------
AtlasBarfed
... How does it do distributed joins? Which just don't scale in the AP sense,
and really can you trust a distributed join in CP?

Unless your data and the data you are joining are strictly shard-local. But
that's also not a distributed join.

~~~
manigandham
It's not a distributed database. It's just a single-node key/value store where
all updates are sent to and then read from a Kafka topic, with relational
querying and schema management handled by various layered Apache projects.

It's a useful tool/model to follow if your data fits on a single node and you
want easy replication across all instances.

~~~
AtlasBarfed
Wow, seems like a lot of work when Postgres can just be setup.

But if someone needs it why not.

~~~
manigandham
This isn't a real product, it's an experiment showing off what you can do.

Kafka is a distributed log, you can use it as replication log if you want and
there are tons of production-ready connectors to all kinds of databases
already. Anyone who needs non-native replication (perhaps across different
databases or sources) should just use those.

------
atombender
Discussed here previously:
[https://news.ycombinator.com/item?id=21048822](https://news.ycombinator.com/item?id=21048822)

