Aggregation seems to happen before filtering out silenced alerts

[kot0dama]

What did you do?

• Set up Alertmanager to group alerts by alertname
• Set up an alert named eg. ServiceDown with a description including some labels, for example: summary: something bad happened in {{ $labels.namespace }}
• Wait for a series to match the alert, for example having label namespace="new-service"
• Silence this alert based on labels included in the summary, here namespace="new-service"
• Wait for a series to match the alert, days later, for example having label namespace="critical-service"

What did you expect to see?

I would expect to get an incident with a description/title set to: something bad happened in critical-service (ServiceDown)

What did you see instead? Under which circumstances?

Instead, incident was grouped with a description/title set to: (ServiceDown) without including the defined summary. I suspect alert grouping to happen before Alertmanager would filter out silenced alerts.

Environment

• System information:

Linux 5.4.0-147-generic x86_64

• Alertmanager version:

Installed from prometheus-alertmanager snap latest/candidate

alertmanager, version 0.26.0 (branch: main, revision: d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)
  build user:       root@lcy02-amd64-066
  build date:       20231230-08:45:02
  go version:       go1.21.5
  platform:         linux/amd64
  tags:             netgo

• Prometheus version:

prometheus, version 2.47.1 (branch: main, revision: c4d1a8beff37cc004f1dc4ab9d2e73193f51aaeb)
  build user:       root@lcy02-amd64-012
  build date:       20231010-03:10:43
  go version:       go1.21.2
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

• Alertmanager configuration file:

Configuration is quite lengthy, but maybe this part is relevant to the bug report:

route:
  receiver: alerts-24x7
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 30m

  routes:
    - matchers:
        - override_group_by = "juju_controller"
      receiver: alerts-24x7
      group_by: ['juju_controller', 'alertname']
      routes:
        - matchers:
          - alert_mute_times="weekend"
          receiver: alerts-weekday
          mute_time_intervals:
            - weekend
            - holidays

• Prometheus configuration file:

Configuration is quite lengthy, not sure which part would be relevant if any.

[TheMeier]

Please provide your receiver config and any relevant templates.

A notification contains multiple alerts, grouped according to the group-by parameters. You should find the description in the annotation of the nested alerts of in commonAnnotations

https://prometheus.io/docs/alerting/latest/notification_examples/#accessing-annotations-in-commonannotations