Hacker News new | past | comments | ask | show | jobs | submit login
Virtual Log-Structured Storage for High-Performance Streaming [pdf] (inria.fr)
48 points by teleforce 11 days ago | hide | past | favorite | 7 comments

While basically almost no real world application needs more performance than what kafka delivers, i love to see research on this topic.

hopefully at some point a more easy to use, non java, "cloud-native" distributed log with decent client apis shows up. (if anyone wants to help building it, please ping me. i am 10 years into building startups and companies with architectures ontop of kafka and pulsar)

Your dream has arrived! Redpanda [1] is a non-Java, ZooKeeper-free streaming platform. It's Kafka compatible, so you can use any Kafka client, of which there are many.

Finally, and perhaps most importantly, it is substantially faster than Kafka.

Disclosure: I work there.


[1] https://github.com/vectorizedio/redpanda

The problem with Kafka is not Java nor ZooKeeper - it is that non-native (non-Java) clients are buggy as hell (last time I touched them), because everybody have to re-implement all the complex logic for each language (often multiple times). Do you have any plans for Redpanda-native protocol/client library?

Why non-java? What do you think is missing that you'd like to build?

Easier to use? Kafka is rather trivial to setup and scale to any size, even more so as they move away from zookeeper

Can't think of a more well behaved java app. Heap usage generally low, never seen GC issues and fast as needed

Is this for write heavy workloads instead of ready heavy workloads? Some of the ideas seem similar to Virtual Consenus in Delos [1].

[1] https://blog.acolyer.org/2020/11/09/delos/

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact