

Fluentd: the missing log collector - kzk_mover
http://blog.treasure-data.com/post/13047440992/fluentd-the-missing-log-collector

======
doryokujin
Cool! Handling each event record as a structured data (JSON) is very
interesting, because it can connect with MongoDB(DocumentDB) seamlessly for
stream output or aggregation. This slide is written about a collaboration with
Fluentd and MongoDB. [http://www.slideshare.net/doryokujin/an-introduction-to-
flue...](http://www.slideshare.net/doryokujin/an-introduction-to-fluent-
mongodb-plugins)

~~~
kzk_mover
awesome slide!

------
abhishektwr
Excellent, and I am going to try it. But I still think using something like
Flume has some advantages mainly because of Hadoop eco-system. For instance
you can plug-in the log data to HBase and use Hive to write to high level
abstracted queries and run on Hadoop. I am guessing but seems like there are
plugins on the way for various systems but not Hadoop. Update: Also Flume can
use any data stream, for instance Twitter stream so not limited to log
Analysis only.

~~~
kzk_mover
Yep, we agree with your opinion CURRENTLY.

We're developing Hoop (REST API for HDFS) plugin, and also Stargate (REST API
for HBase). It requires additional setup at the infra-side, but still very
useful and performs well.

You can feed Twitter stream by writing in_twitter plugin. That will 30 lines
of Ruby code :-)

------
maratd
Err... what? Am I missing something? Just use rsyslog and transfer the logs
over TCP/IP to a central server. No need to re-invent the wheel.

~~~
viraptor
It's more about log analysis than just transferring the files over. Apart from
splunk there aren't many projects which handle this well.

It also seems to have many input plugins apart from standard syslog. And
that's great. I think we're way past the time where lines of text just don't
work well enough. First you format your data to text, then you want to get
your original data back from that text by parsing... seems like a wasted
effort.

(although there's graylog2 with GELF)

------
moaikids
coooool!!

