
Ask HN: Logstash vs. Kafka? - albertlie
Hi all,<p>I&#x27;m trying to build realtime data infrastructure for logging. For ingestion layer, I&#x27;m thinking about using Kafka &#x2F; Logstash for logging ingestion layer and then after that I can store it in any database and I can easily change the store without changing the ingestion layer in the future.<p>Any experience using logstash or Kafka in production?<p>Also, additional question is, I concern a lot for missing data case  when we&#x27;re sending data to logstash or kafka using kind of lightweight shipper like filebeat? Any experience for handling missing data at scale?
======
dozzie
Are you really ready to forego hundreds of megabytes of RAM for merely log
shipping? Fluentd could be a cheaper alternative to JVM-based routers.

Also, what exactly is your question?

> Any experience for handling missing data at scale?

For _logs_? Missing logs _don 't matter_ (unless they're a required audit
data). Your system should be prepared not to fall apart on missing hours or
days of logs, similar to how it should treat missing metrics and other
monitoring data.

And what volume is "at scale"?

~~~
albertlie
Hi Dozzie,

Thanks for the answer.

Just for clarification,

1\. For my case (yea, they are kind of required audit data), I need the log to
be able to not loss when sending to logstash or kafka, probably this should be
handled in client side by buffering the data before sending to logstash?

2\. For volume "at scale", my point is just for clarify whether this logstash
could be good to be used for production for big volumes. Maybe can be like
100GB per day

~~~
dozzie
> 1\. For my case (yea, they are kind of required audit data)

What kind and required by whom? Most of such things are "if you have all,
good, if half an hour is missing, we'll do without those", not something akin
to transaction data in relational database where you cannot lose even a single
byte.

> I need the log to be able to not loss when sending to logstash or kafka

What is "not to lose logs"? Is losing a single record OK? Is losing 1% of logs
OK? How long network problems should be tolerated? What to do if such network
problems are longer than anticipated and disk buffer is running out of space?

You have said _virtually nothing_ about what your logs consist of and what
kind of reliability you require from them.

~~~
albertlie
Hi Dozzie,

That's great question

> What kind and required by whom? Most of such things are "if you have all,
> good, if half an hour is missing, we'll do without those", not something
> akin to transaction data in relational database where you cannot lose even a
> single byte.

So we will do log analysis from the data (counting, sum of the amount from the
data, etc). So ideally need 100% precision and we will do the analysis in
daily basis

> What is "not to lose logs"? Is losing a single record OK? Is losing 1% of
> logs OK? How long network problems should be tolerated? What to do if such
> network problems are longer than anticipated and disk buffer is running out
> of space?

In my case, losing single record is significant enough because we will do
calculation for those logs. For the network tolerance, I think that's very
good question, I'll take that as my consideration too. But the network
problems that I mean before is the intermittent one like < 1 minutes and with
assumption the data buffered is not until making the server running out of
space

------
htn
We're using Kafka as a log delivery platform and are quite happy with it.
Kafka by nature is highly available and can be scaled quite trivially with the
log load by adding new cluster nodes.

We've decided to use journald for storing all of our application data. We pump
the entries from journald to Kafka by using a tool that we open sourced:
[https://github.com/aiven/journalpump](https://github.com/aiven/journalpump).

From Kafka, we export the logs to Elasticsearch for viewing and analysis. Some
specific logs are also stored in S3 for long term storage for e.g. audit
purposes.

