Rancher monitoring - Enable external metrics?

[josh-at-straker]

Hey all. I can't seem to get to the slack at the moment, so posting here.

I am having some trouble getting the external metrics apis running in our RKE2 cluster.

Back story:
I've setup a new endpoint for Prometheus to scrape, I have added my extra scrape information to the rancher-monitoring helm chart and now can see my extra metrics.

I've then gone and added a config snip to the the prom-adapter to expose my new metrics for further use (eventually as a option in our HPAs).

- seriesQuery: '{__name__="redis_stream_group_messages_pending", job="straker-uat-redis-stream-exporter", group="verify_consumer",  stream="verify:job:evaluate:file:fa"}'
  resources: {}
  name:
    as: "redis_stream_group_messages_pending_verify:job:evaluate:file:fa"
  metricsQuery: '<<.Series>>{<<.LabelMatchers>>}'

While the config has added correctly to the adapter, it does not show in the v1beta1.custom.metrics.k8s.io output. Further reading online shows I've set this up as an external source, so I should be seeing the metric under v1beta1.external.metrics.k8s.io instead.

However, in my cluster, I can not get v1beta1.external.metrics.k8s.io to show:

kubectl get apiservices | grep metrics
v1beta1.custom.metrics.k8s.io          cattle-monitoring-system/rancher-monitoring-prometheus-adapter   True        36d
v1beta1.metrics.k8s.io                 kube-system/rke2-metrics-server                                  True        678d

Do I need to do something special to enable this api set with the rancher monitoring helm chart, or change something in my cluster?

Cheers.

[josh-at-straker]

I've solved this, turns out I needed to spend some more time reading the prometheus adapter helm chart.

Only when a external metric is added this way (below), is the external metrics API setup for the cluster. No other method appears to work.

TLDR; for setting up a new external metric:

prometheus-adapter:
  enabled: true
  prometheus:
    port: 9090
    url: http://rancher-monitoring-prometheus.cattle-monitoring-system.svc
  psp:
    create: true
 ### Add the below
  rules:
    default: true
    external:
      - seriesQuery: '{__name__="redis_stream_group_messages_pending", job="straker-uat-redis-stream-exporter", group="verify_consumer",  stream="verify:job:evaluate:file:fa"}'
        resources:
          namespaced: false
        name:
          as: "redis_stream_group_messages_pending_verify:job:evaluate:file:fa"
        metricsQuery: '<<.Series>>{<<.LabelMatchers>>}'

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/" | jq 

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "redis_stream_group_messages_pending_verify:job:evaluate:file:fa",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

Cheers.