Hacker News new | past | comments | ask | show | jobs | submit login
Some thoughts on Prometheus Alertmanager's alert reminders (utcc.utoronto.ca)
24 points by zdw on Jan 5, 2023 | hide | past | favorite | 10 comments



The root problem here is not a remind interval, but "email alerts".

Either something should file a bug to look at "some time later", or it should start ringing your phone.

Email alerts are just spam.

For both bugs and pages the reminder is good. A bug should get an update essentially saying "yes, this is still looking bad" (not create a new bug), and if it was important enough to page you the first time, then (unless you silenced the alert) it's important enough to remind you by ringing your phone again.


This is natively supported in PagerDuty, OpsGenie, etc. Routing, time-based suppression, alert reminders, etc. I’ve tried to implement all of that in Alertmanager itself, and it’s much easier to just swipe the card.

As for the dashboard, there’s an Alertmanager data source for Grafana that’s super easy to set up. If you have Grafana already I would just go ahead and do it since it takes maybe ten minutes.


> This is natively supported in PagerDuty, OpsGenie,

What is a free self-hosted alternative to these two?


Grafana

It now uses the Prometheus Alertmanager natively as its alert manager. And if you were using something like Mimir instead of Prometheus then you can seamlessly use Grafana against that Alertmanager too. Grafana is comfortable with any number of Prometheus compatible alert managers since the last major version.


Alertmanager.


GP's claim was that it was not


I found myself nodding along with the author's musings and don't have much to add. I just want to say that I love their site. I'm putting together a personal site of my own and I want to steal some of their ideas, especially the "View Source" page tool at the bottom.


looks like its some form of Markdown


Agreed. Though I meant more the functionality of being able to click on it.


Perhaps severity should also influence how often an alert is repeated. And during a holiday break where no one is available to react to incidents, a alert silence should be active until people are back.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: