
Kafka as database – is it safe? - walrus1066
Specifically, how strong are the durability &amp; backup guarantees?<p>The setup is:<p>-kafka is the underlying source-of-truth for the data.<p>-querying is done on caches (I.e. Redis, ktables) hydrated from kafka<p>-Kafka configured for durability (infinite topic retention, 3+ replication factor etc)<p>We are not allowed to lose data <i>in any circumstances</i><p>Is the proposed setup nuts?<p>In <i>theory</i> the replication should guarantee durability &amp; resiliency. Confluent themselves encourage the above pattern.<p>The only flaws I can think of, are:<p>- cache blows up and needs to be rehydrated from scratch<p>-broker disk gets wiped&#x2F;corrupted -&gt; kafka rebalance, resulting in prolonged downtime if topics contain mountains of data<p>Has anyone run and battle tested this kind of setup in production? I.e. encountered disk corruption, brokers going down, but still retain data?<p>My gut feel is, this is a bad setup, because kafka wasn&#x27;t originally designed for RMDBS levels of durability, but can&#x27;t point to a concrete reason why this would be the case.
======
borplk
Can you talk more about the use case?

My question would be, why use that setup if you don't have to?

That setup is not something I would reach for unless I really really had to
(and when you think you really really have to, check again, then check again,
you probably don't really really have to).

How about a RDBMS like Postgres setup with streaming synchronous replication
and durability-tweaked parameters?

~~~
walrus1066
In my opinion, using RMDBS as source of truth makes more sense, because these
systems were optimised for reliability, durability & data retention.

To be completely honest, some have drunk the proverbial Confluent koolaid,
with kafka being the hammer to hit all nails.

I just want to communicate the risks & tradeoffs with taking this approach.
Problem is, the theory sort-of holds up, if configured correctly, kafka
_should_ be as reliable/durable as PG/SqlServer/MySQL cluster. But I can't
find any material on companies experience running kafka this way, I.e. 'we ran
this setup for a year in prod, one of our brokers disks got corrupted, but we
were able to get back with no data loss, via X,Y,Z.' or 'we couldn't recover
and had to accept data loss'.

EDIT: in short, are the claims made by this article reliable:
[https://www.confluent.io/blog/okay-store-data-apache-
kafka/](https://www.confluent.io/blog/okay-store-data-apache-kafka/)

