Sending e-mail or displaying a set of charts or a status table is simple.
Allowing to collect, collate and aggregate the data (metrics and events) in
arbitrary way, also as an afterthought, is what monitoring system should do.
With virtually everything on the market, when a need for any processing not
anticipated by monitoring system author arises, one needs to write much stuff
outside said system.
We need less systems resembling invoicing systems and more systems resembling
general purpose databases.
This is why monitoring still sucks.
(me) Tell me "What other abnormal activity is correlated to that peak in network activity?"
(monitoring) "Sure, servers srv1, srv5 and srv7 CPU activity peaked at the same time, and for the following 30 seconds, server syslog12 disk IO went crazy, mostly with repeats of the log message referenced below."
I admit I do very little systems/network administration work, so maybe there's something I'm missing.
Also, you say I can query against such database. But does the monitoring
system allow me to query that database? I have no documentation for it.
Most of the time I can't easily fill events generated out of that back to the
database. And at last, current so-called "state of the art" monitoring systems
don't facilitate running custom queries, so I would need to write something
totally external just to run the query.
Graylog2 and Riemann go a little in this way, but they stopped way, way too
soon to be an answer to current state of monitoring systems.
Observium's niche is user-friendly software for comprehensive monitoring of networks (and some server/appliance things). Setup is as easy as Wordpress and it will monitor pretty much everything on a network (ISP/carrier). Adding a device is pretty much one click, add hostname.
In exchange it expects things like having resolvable hostnames for devices and standardized port descriptions.
Compare this with Nagios or Cacti where you have to configure checks/graph collection for each thing you care about. The skill level required to set them up is a bit higher than Observium but still free.
Alternatively you have commercial software like Orion or Intermapper where setup may be easier but you have to pay for every metric you collect.
Because of this Observium gets a stream of vocal small MSP/ISP-type people who demand support and aren't willing to pay for it. I believe Obserivum went to a pay-for-access model in part to keep this under control.
I use Observium, pay for the subscription (~$200/yr is really trivial for this), and have contributed patches (that were committed) that fixed bugs and added device support (without any compensation aside from it making $DAYJOB easier). From the WISP community alone I saw probably half a year of "I won't buy a subscription until you support X device/X pet feature" from people who often came off as the client from hell in how they said it.
So, yeah, I don't like that I end up with a bit of anxiety over the reception my patches will get but I see the occasionally rough reaction as a immune response from a community that's under stress.
…I'm not pointing in any direction… ducks
I do agree with dozzie though and that's something LibreNMS hasn't yet resolved which is being able to correlate data from different devices / systems to give you the one true answer of what's just gone wrong. I'd like to think we could get closer to being able to do that with more people contributing + some fundamental changes to the software.
That's more a job for SNMP clients anyway. Actually, just look at Nagios/Icinga :) It's a very mature and well-rooted infrastructure monitoring solution.