Review "group_interval" to differentiate new alerts in group from resolution

[f-ld]

Originally posted on grafana/grafana and posting here as reported to be:

I think your use-case is a valid one, however this sounds like a discussion that should be had in https://github.com/prometheus/alertmanager

Copy paste of the original message:

---

In notifications policies, my understanding of group_interval is the amount of time to wait before sending any notification for an alert that changed.

Let's assume I have an a notification policy configured like this:

    group_by:
      - alertname
    group_wait: 10s
    group_interval: 5m
    repeat_interval: 8h

And an alert with same name but being triggered for various pods (I am running in k8s).

If the alerts gets triggered for pod A only, then within 10s I will get a notification with pod=A in labels.
If then pods B and later pod C are triggering the alert, then no notification will be sent for then within 5 minutes. Only after 5 minutes it will be sent. In my case, I would like this 5m setting to be higher, like 1h to avoid spam of alerts when the list of pods is changing quite a lot for some time. So I would set group_interval: 1h.

But instead of the above new pods added to the alert I have the use case of the alert being resolved, it will only send a notification after group_interval: 5m or group_interval: 1h depending on what I have configured. And I would like this to be lower.

Hence my requests:

• first, is my understanding correct?
• then, can we have group_interval split into group_interval and resolution_interval.

Indeed, I understand some might not want to resolve too quickly to avoid up/down effect in case of instabilities.
But I have a metric in my case that is triggered during rollout restart and that has many updates for some time (maybe 30 to 60 minutes) and get spammed with notifications that I have to silence. I would prefer having one ... but be notified asap (understand 1 minute for example) if the alerts gets resolved.

I hope the above makes sense, let me know.

---

Thank you