Using k8s labels in prometheus rules expr

[vlogachev-bd]

Hello,

I am new to Prometheus so I would ask for some advice.

I am trying to use custom labels from k8s deployment in the prometheus rules expression.

Here is my setup:

k8s labels:

labels:
app: nginx-test
ignore-desired-replicas: "true"
ignore-ram-usage: "true"
ignore-cpu-usage: "true"
ignore-container-restarts: "true"
team: xxx

kube-state-metrics deployment.

args:
- '--metric-labels-allowlist=pods=[team, ignore-ram-usage, ignore-cpu-usage, ignore-container-restarts],deployments=[team, ignore-desired-replicas],persistentvolumeclaims=[team]'

Prometheus rule
expr: kube_deployment_labels * on (deployment, namespace) group_right(label_team) ((kube_deployment_status_replicas_available / kube_deployment_spec_replicas) < 1 )

I also tried to add group_right(label_team, ignore-desired-replicas) and ( (label_team, label_ignore-desired-replicas), but I am not even able to startup the container, I assume due to the fact that it does not recognize the label.

It looks to me that I am making a mistake somewhere in the kube-state-metrics configuration.

Currently, when I receive an alert I can see that it groups only be team and it is ignoring my second label ignore-desired-replicas.

Any thoughts, folks?

[k8s-ci-robot]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.