Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[collector] failing to add additional groups in prometheus rules #1428

Open
kkantonop opened this issue Nov 21, 2024 · 0 comments
Open

[collector] failing to add additional groups in prometheus rules #1428

kkantonop opened this issue Nov 21, 2024 · 0 comments
Labels
bug Something isn't working chart:collector Issue related to opentelemetry-collector helm chart help wanted Extra attention is needed

Comments

@kkantonop
Copy link

kkantonop commented Nov 21, 2024

I'm trying to add an additional custom rule on top of the default rules in prometheus rules.

I'm using the 0.108.0 chart version with the following prometheusRule values:

prometheusRule:
  enabled: true
  groups:
  - name: addtionalCollectorRules
    rules:
    - alert: ExporterQueueFull
      expr: 100 * sum by (exporter) (otelcol_exporter_queue_size) / sum by (exporter) (otelcol_exporter_queue_capacity) > 90
      for: 2m
      labels:
        severity: critical
      annotations:
        description: Queue of exporter {{ $labels.exporter }} is more than 90% full
  defaultRules:
    enabled: true

The helm commands fail (when constructing the templates) with the following error

$ helm template -f values.yaml open-telemetry/opentelemetry-collector --version 0.108.0
Error: YAML parse error on opentelemetry-collector/templates/prometheusrule.yaml: error converting YAML to JSON: yaml: line 25: did not find expected key

Use --debug flag to render out invalid YAML

Running it with the debug flag indeed reveals that the additional group is one indentation level to the right

# Source: opentelemetry-collector/templates/prometheusrule.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: release-name-opentelemetry-collector
  namespace: default
  labels:
    helm.sh/chart: opentelemetry-collector-0.108.0
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "0.111.0"
    app.kubernetes.io/managed-by: Helm

    release: "kube-prometheus-stack"
spec:
  groups:
    - name: addtionalCollectorRules
      rules:
      - alert: ExporterQueueFull
        annotations:
          description: Queue of exporter {{ $labels.exporter }} is more than 90% full
        expr: 100 * sum by (exporter) (otelcol_exporter_queue_size) / sum by (exporter)
          (otelcol_exporter_queue_capacity) > 90
        for: 2m
        labels:
          severity: critical
  - name: collectorRules
    rules:
    - alert: ReceiverDroppedSpans
      expr: rate(otelcol_receiver_refused_spans[5m]) > 0
      for: 2m
      labels:
        severity: critical
      annotations:
        description: 'The {{ $labels.receiver }} receiver is dropping spans at a rate of {{ humanize $value }} per second '
        runbook_url: 'https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/monitoring.md#receive-failures'
...

Should I pass a different value or different format to groups?

After testing locally, I can create the template without error after changing the line here from

{{- toYaml .Values.prometheusRule.groups | nindent 4 }}

to

{{- toYaml .Values.prometheusRule.groups | nindent 2 }
@TylerHelmuth TylerHelmuth added bug Something isn't working help wanted Extra attention is needed chart:collector Issue related to opentelemetry-collector helm chart labels Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working chart:collector Issue related to opentelemetry-collector helm chart help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants