Help with deadman switch

[lonelyadmin]

Looking for some help with setting up a dead mans switch. I'm still wrapping my head around alertmanger. I've been implementing rules from here if they fit my environment. I'm trying to setup a deadman's switch, but I can't figure out how to invert the alert for this particular one. Should I be trying to setup a route that discards particular alerts, or should that be done at the receiver? If so, how? I just want to get an alert when this alert isn't firing.

Also, for an always firing alert, do you just silence it for an incredibly long time? Or can you negate that too somehow? I'd like to have the firing alert count always be zero rather than one.

Thanks in advance.

My setup:

prometheus rule:

      expr: 'vector(1)'
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: Prometheus AlertManager E2E dead man switch (instance {{ $labels.instance }})
        description: "Prometheus DeadManSwitch is an always-firing alert. It's used as an end-to-end test of Prometheus through the Alertmanager.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

My relevant alertmanager config:

  group_by: ['alertname', 'cluster', 'service', 'instance']                                                                                                                                                                                                     
  group_wait: 30s                                                                                                                                                                                                                                               
  group_interval: 5m                                                                                                                                                                                                                                            
  # How often alerts are sent                                                                                                                                                                                                                                   
  repeat_interval: 3h                                                                                                                                                                                                                                           
  receiver: DEFAULT_EXCHANGE_STARTTLS                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                
  routes:                                                                                                                                                                                                                                                       
     - matchers:                                                                                                                                                                                                                                                
         - alertname=~"PrometheusAlertmanagerE2eDeadManSwitch"                                                                                                                                                                                                  
       group_wait: 0s                                                                                                                                                                                                                                           
       group_interval: 1m                                                                                                                                                                                                                                       
       repeat_interval: 50s                                                                                                                                                                                                                                     
       receiver: 'EXCHANGE_STARTTLS_DEADMAN'                                                                                                                                                                                                                    
     - matchers:                                                                                                                                                                                                                                                
         - alertname!~"PrometheusAlertmanagerE2eDeadManSwitch"                                                                                                                                                                                                  
       group_wait: 10s                                                                                                                                                                                                                                          
       #continue: true                                                                                                                                                                                                                                          
       receiver: 'EXCHANGE_STARTTLS'                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                
inhibit_rules:                                                                                                                                                                                                                                                  
  - source_matchers: [severity="critical"]                                                                                                                                                                                                                      
    target_matchers: [severity="warning"]                                                                                                                                                                                                                       
    equal: [alertname, cluster, service]                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                
receivers:                                                                                                                                                                                                                                                      
  - name: 'DEFAULT_EXCHANGE_STARTTLS'                                                                                                                                                                                                                           
    email_configs:                                                                                                                                                                                                                                              
      - smarthost: 'mail.domain.com:587'                                                                                                                                                                                                                        
        to: '[email protected]'                                                                                                                                                                                                                              
        send_resolved: true                                                                                                                                                                                                                                     
        tls_config:                                                                                                                                                                                                                                             
          insecure_skip_verify: true                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                
  - name: 'EXCHANGE_STARTTLS'                                                                                                                                                                                                                                   
    email_configs:                                                                                                                                                                                                                                              
      - smarthost: 'mail.domain.com:587'                                                                                                                                                                                                                        
        to: '[email protected]'                                                                                                                                                                                                                              
        send_resolved: true                                                                                                                                                                                                                                     
        tls_config:                                                                                                                                                                                                                                             
          insecure_skip_verify: true                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                
  - name: 'EXCHANGE_STARTTLS_DEADMAN'                                                                                                                                                                                                                           
    email_configs:                                                                                                                                                                                                                                              
      - smarthost: 'mail.domain.com:587'                                                                                                                                                                                                                        
        to: '[email protected]'                                                                                                                                                                                                                              
        send_resolved: false                                                                                                                                                                                                                                    
        tls_config:                                                                                                                                                                                                                                             
          insecure_skip_verify: true ```

[lonelyadmin]

Should this be asked elsewhere? Or should I just ask better questions? :)

[simonpasquier]

Also, for an always firing alert, do you just silence it for an incredibly long time? Or can you negate that too somehow? I'd like to have the firing alert count always be zero rather than one.

By definition the deadman snitch alert (or Watchdog) needs to be always firing. It's usual to set the severity to something other than critical. See here for instance:

https://github.com/prometheus-operator/kube-prometheus/blob/c936a999acdbee7b1134bcf4be230e458d3ed9cd/manifests/kubePrometheus-prometheusRule.yaml#L27-L40

You can also find an example of route + receiver configuration at https://github.com/gouthamve/deadman