We use Kafka a lot within the company I work for. I think it's great.
The only thing I miss is fast lookups based on some key and/or the ability for subscribers to only receive messages for certain keys.
That’s most likely the trade off between the queue and a log. The property of the log is having to look at every message.
The prefixes can be somewhat done with a custom partitioner, the lookup requires another technology. The only way to know the final value under the key is to look at the complete log.
This should be labelled as Show HN. One comment - towards the end you say
> You might think that Kafka would be using queue data structure internally. It’s not true. Kafka uses a “log” data structure. It is a persistent data structure which allows only appends, no editing, no deletion. In detail, we will cover this some other day.
Almost immediately followed by:
> Ok, I have a confession . I lied. A Kafka topic is not just a single queue. It’s a combination of queues which helps kafka scale. Every queue is called partition.
> Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead.
https://engineering.linkedin.com/distributed-systems/log-wha...
Reading it was an aha. At its core, the log is the simplest thing that might work in a lot of cases.