
My Philosophy On Alerting (2014) - pmoriarty
https://linuxczar.net/sysadmin/philosophy-on-alerting/
======
mikece
“...over-monitoring is a harder problem to solve than under-monitoring.”

Amen! Too many times I’ve seen the assumption that if we’re logging something
then we need to monitor it, and if we’re monitoring then we need to alert on
it. And the end result is an operations alert email address getting 30 or 40
emails an hour that serve little more than at assure the systems are still
running and increasing the chances that important alerts will get buried and
ignore or not noticed at all.

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=8450147](https://news.ycombinator.com/item?id=8450147)

------
badrabbit
Interesting how so little of this can be applied to security alerts. E.g.:
most security alerts can't be actioned.

~~~
tempguy9999
A most interesting statement. If security is reported broken or compromised,
it's very actionable, surely?

~~~
badrabbit
No, security alerts tell you something might be wrong. You have to then
perform some analysis tasks and determine if it is a true or false positive
alert. Even if it is a true positive you may not have enough info to
action,you might continue monitoring or dismiss the threat. You might have to
determine the cost of further investigstion and remediation vs security risk.

For example in a past life I was in a position where malware traffic will be
detected from some locations but the infected device can't be located. Or some
overt webapp scanning is happening,I want to know about it and make sure it
isn't excessive but it's not worth the effort of blocking.

You don't really action security alerts and events,you action security
incidents(which are security events/alerts confirmed to be true positive).
With infrastructure monitoring if you see an alert that means something is
wrong,the alert is an incident. Been on the responding side of both types of
alerts.

Incident response is very interesting,most plans hardly survive the
battlefield.

