

Running Apache Kafka at Scale - boredandroid
https://engineering.linkedin.com/kafka/running-kafka-scale

======
robalfonso
This is very timely. We are just starting to work on a new centralized logging
system using logstache->kafka->elasticsearch->kibana, we may also use kibana
for some other tasks as well. Anyone have any experience using that setup?
Pros/Cons?

We used graylog for a while but ran into some issues with back pressure on
elastic search (hence kafka).

~~~
toomuchtodo
Look into the new Graylog (v1.0). It was recently released, and they use Kafka
internally now for buffering. We're sending thousands of msgs per second to
Graylog without issue.

~~~
waitwaitwhay
According to their documentation, the data storage for log events is
Elasticsearch and if the Elasticsearch data is lost then the logs are gone.
Consideribg that ES is not a database and may lose data this sounds a bit
scary to me.

~~~
_up
My understanding is that appending only is fine. So if your use case is
logging elasticsearch is totatly fine. You probably can't compare it to the
robustness of MySQL & PostreSQL. But most NoSQL are not known to be that
robust either.

------
felipesabino
I heard comparisons with other technologies before, like RabbitMQ [1] [2] but
I would love to see an feature/performance comparison between kafka and other
similar solutions, specially with cloud based services like the new kid on the
'Google Pub/Sub' [3]

[1] [http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-
durable...](http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-durable-
messaging-with-good-query-features/answer/Stuart-Charlton-1)

[2]
[https://youtu.be/MA_3fPBFBtg?t=35m27s](https://youtu.be/MA_3fPBFBtg?t=35m27s)

[3] [https://cloud.google.com/pubsub](https://cloud.google.com/pubsub)

