fluentd is great.
You can setup forwarding nodes, that relay logs to one or mutiple masters that then persists into whatever layer(s) you want.
Tolerance and failover baked in.
Tons of connectors and best of all docker logs driver is built ships with docker so almost zero setup to get your container logs to fluentd.
Also works nicely with kubernetes too!
I haven't used it in maybe 3 years or so, so some of this could be misremembered a bit, but I didn't have a great experience with fluentd.
Trying to do much customization was kind of painful, the config file structure is kind of confusing and the docs were sparse and differ depending on the plugin you're using, and there's no validation of the config so if you have any of the arguments slightly wrong it'll fail silently.
The modularity of routing and filtering logs seems like it would be great, but it turns out not to be all that flexible, you really have to follow the framework's preconceived idea of how the processing pipeline should work. I forget the details but we were trying to do some processing on the logs, like ensuring they're valid json and adding a few fields to them based on other metadata source, and it would have required writing our own plugin. In other ways it felt too modular for its own good, like it's up to individual plugins to implement anything they need and the core doesn't seem to provide you with much. Even things like an error handling or retry framework are not built in, so if a plugin throws an unexpected exception it's not handled gracefully. One plugin we were using would just stop sending logs until we restarted the fluentd process if it ever got an http failure from being temporarily unable to connect to the api, because they hadn't built in their own retry/recovering mechanism. Granted this was for a pretty new logging SAAS service we were trying out, if you're sticking with the basics like files and syslog it's probably more robust, but at that point you also might not need the flexibility promised by a tool like fluentd.
Despite their advice not to write a lot of Ruby code, the fluentd plugins are Ruby code, so I ended up just writing one big custom plugin with all the rewrite/routing logic in it. It was very simple to do, and so much easier to read than the convoluted fluentd config files you end up with to do anything mildly complicated with filtering.
