You're right to be concerned about complexity: simple things are easy to understand, easy to predict, and easy to change.
That said, I think you'll find many of the ideas in Riemann to be radically simple. The config file is just a Clojure program. Streams are just functions that take events. Events are just maps of keys to values. Everything is an event: there is no concept of a first-class host or service, no need to update the config when you add a host, and no poller loops.
Riemann tries to draw strong boundaries between the different layers of monitoring. It speaks a simple network protocol and interoperates with other systems for event collection, visualization, alerting, and storage, instead of building in those systems. In many ways Riemann is defined not by what it includes, but by what it leaves out.
That said, there's a lot of work required to make simple abstractions behave correctly, especially around IO and error handling. Wherever possible I try to draw clear internal boundaries to isolate this complexity, but it's still there. If you have specific complaints about code or interfaces which seem too complex to you, I'd be happy to try and explain or change them.
The tone around here is often negative, but I want to give you a huge compliment: you understand simplicity.
Reducing hairy things to simple abstractions can save weeks of work in a matter of minutes. It is the single most powerful programming technique I know of. And I'm going to seriously consider switching a bunch of stuff over to Riemann.
hi, this project looks really neat! i'm evaluating icinga web + pnp4nagios at work, but this could be a viable alternative. cheers and glad you put this out there :-)
i see that you use Protocol Buffers. from google's page, it seems like they only work with C++, Java or Python. now this could be a problem for us. what if i want to pull events from a bash script, a delphi gui app, SNMP, Dell idrac interface, or any other event? do they have to interface over Protocol Buffers, or am i missing something here? would i have to write a glue layer?
and what about, if you have two seperate networks, and want one server to forward data for its entire lan to the other server, to process and graph them?
There are protobuf bindings for many languages, though I hear node.js was a bit of a pain. Check the clients page and see if the language you need is there; I can help you build one if not.
For pulling from other tools, I usually write a little daemon to poll and relay the data. See riemann-tools for a collection of existing tools to do just that; and you can require it as a library to write your own in just a few lines of ruby.
Forwarding between servers is built in; it's easy to aggregate events in hierachies for large-scale analysis.
That said, I think you'll find many of the ideas in Riemann to be radically simple. The config file is just a Clojure program. Streams are just functions that take events. Events are just maps of keys to values. Everything is an event: there is no concept of a first-class host or service, no need to update the config when you add a host, and no poller loops.
Riemann tries to draw strong boundaries between the different layers of monitoring. It speaks a simple network protocol and interoperates with other systems for event collection, visualization, alerting, and storage, instead of building in those systems. In many ways Riemann is defined not by what it includes, but by what it leaves out.
That said, there's a lot of work required to make simple abstractions behave correctly, especially around IO and error handling. Wherever possible I try to draw clear internal boundaries to isolate this complexity, but it's still there. If you have specific complaints about code or interfaces which seem too complex to you, I'd be happy to try and explain or change them.