
Rochefort – Poor Man's Kafka - booknomads
https://github.com/jackdoe/rochefort
======
saryant
I've run Kafka at large scale. I've also seen even larger scale attempts to
replace it.

Just use Kafka. Seriously, it's rock solid and is practically lingua franca in
backend architecture these days. Everyone understands it and every data
processing framework or service supports it.

Kafka is much, _much_ more than just distributed pub/sub. It's disk cache
optimizations _alone_ make rolling your own a terrible idea.

~~~
kylecordes
This is a principle our industry implements poorly. It often seems like each
new generation (for very small values of the word generation) must reinvent
the same thing. Perhaps because the old thing was too complex to understand
immediately... complexity driven by the needs of the underlying problem...
complexity which the new implementation will inevitably obtain if it survives
long enough and becomes popular enough that anyone cares.

(That said, I'm highly in favor of innovation of most any kind; building new
things is great, if the new thing has some plausible innovation over the old
thing!)

~~~
borplk
The industry itself feeds this cycle by rewarding people for creating new
projects and demanding shiny github projects.

------
luhn
If you're looking for a log that's not Kafka, also worthwhile to check out
Redis Streams, which will be included in the upcoming Redis 5.0 release.

[http://antirez.com/news/114](http://antirez.com/news/114) (I think the API
has changed a bit since this blog post, but the concepts and capabilities are
the same.)

------
voxadam
On a somewhat related note there is a link implementation of Kafka written in
Go called Jocko.[0]

[0]
[https://github.com/travisjeffery/jocko](https://github.com/travisjeffery/jocko)

~~~
tbrock
This is actually interesting because Kakfka’s is great and the protocol makes
sense but I want a binary to run so I don’t have to deal with the java
ecosystem.

------
jitl
Backstory and caveats:

> _Losing Data and NIH_

> You can lose data on crash and there is no replication, so you have to
> orchestrate that yourself doing double writes or something.

> The super simple architecture allows for all kidnds of hacks to do
> backups/replication/sharding but you have to do those yourself.

> My usecase is ok with losing some data, and we dont have money to pay for
> kafka+zk+monitoring(kafka,zk), nor time to learn how to optimize it for our
> quite big write and very big multi-read load.

> Keep in mind that there is some not-invented-here syndrome involved into
> making it, but I use the service in production and it works very nice :)

~~~
549362-30499
I'm scratching my head about "we dont have money to pay for
kafka+zk+monitoring(kafka,zk)". Kafka and Zookeeper are both open source. As
are monitoring and alerting tools such as Prometheus. Surely the hosting and
storage costs are similar. So what does this project offer its creator, other
than a great deal of infrastructural debt and all the latent bugs of a roll-
your-own solution that lacks a community?

~~~
matt_wulfeck
I'm assuming he means money as in time, to install, configure, optimize, and
monitor those distributed systems.

~~~
549362-30499
Even if that's the case, deploying and scaling a Kafka cluster is something
that hundreds of companies have figured out and publicly written about. It's
something that you _can_ hire an experienced engineer to fix. When this thing
runs into problems, they will be all new ones.

~~~
bonesss
These days you can also find kubernetes scripts that handle the Kafka
installation, setup, routing, etc.

That just leaves the "simple work" of administering and tuning which, as you
pointed out, is competence that's steadily growing in the industry.

------
bognition
I'm always very curious about the backstory of projects like this. Without
that backstory there is very little chance I'd try out something like this.

Ideally the read me would explain why Kafka didn't cut it, why the trade offs
the authors made were worth it (in this case), and why I did consider using a
this system.

Sadly I don't have enough time to read an entire repo of code to try and
figure these things out.

~~~
LogicX
Atleast read the bottom of the README where some of this is covered

------
agnivade
Question to OP - Did your team check out NATS
([https://nats.io/](https://nats.io/)) ? What are your thoughts on that ?

------
z0r
this repository appears to be just a hair over a week old, so i am skeptical
even of "I use the service in production and it works very nice". fun project
i'm sure, but if i felt like breaking the rules and engaging in a little NIH
of this sort - i'm not sure i'd choose HTTP (or any other network protocol) as
the hub to build it around

~~~
amerine
I don’t disagree with you, buts it’s also possible it was modeled/extracted
from something non-public.

------
miguelrochefort
How did you pick the name?

~~~
joefreeman
"Rochefort Trappistes 10 is my favorite beer and I was drinking it while doing
the initial implementation at sunday night"

