Setting up Keda for autoscaling fails with error: Failed to create new metrics collector

[dcfranca]

I have created it under Q&A, but as I didn't get any response I'm creating it here as well for more visibility

I'm trying to setup Keda for the first time with a sample application scaling based on AWS SQS queue

Based on my understanding of how Keda works:

• Keda only handles the scale from 0 to 1 and from 1 to 0
• After scale to 1 it passes the control to K8s HPA
• K8s HPA fetches the metrics from the metrics server
• Metrics server uses kube-metrics-adapter to get external metric

I have set the keda namespace
It contains 2 pods running on it: keda-operator and keda-metrics-apiserver

There is a ScaledObject defined to handle the scaling of a sample application scaling based on an SQS Queue

The scaled object is as follows:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: demo-sqs-keda-scaledobject
  namespace: mynamespace
spec:
  scaleTargetRef:
    name: demo-sqs
  minReplicaCount: 0 # We don't want pods if the queue is empty
  maxReplicaCount: 5 # We don't want to have more than 5 replicas
  pollingInterval: 30 # How frequently we should go for metrics (in seconds)
  cooldownPeriod: 120 # How many seconds should we wait for downscale
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/XXXX/my-queue
        queueLength: "10"
        awsRegion: "eu-west-1"
        identityOwner: operator

I have created an IAM role and I'm using its role to attach it to the operator pod and metrics-server pods

The Keda-managed scaling process works fine, the pods manage to scale from 0 to 1 and from 1 to 0
However, that is all we get

When it delegates the scaling to the HPA it fails to collect the metrics with the following warning(that looks like an error) coming from the kube-metrics-adapter

time="2023-03-14T12:16:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"HorizontalPodAutoscaler\", Namespace:\"product\", Name:\"keda-hpa-demo-sqs-keda-scaledobject\", UID:\"XXXXXX\", APIVersion:\"autoscaling/v2beta2\", ResourceVersion:\"XXXXX\", FieldPath:\"\"}): type: 'Warning' reason: 'CreateNewMetricsCollector' Failed to create new metrics collector: no plugin found for {External {s0-aws-sqs-demo-my-queue &LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: demo-sqs-keda-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}}}"

Apparently it miss a plugin to collect the metric (?)
I can't find anything on Keda documentation regarding plugin installation or some extra setup needed

I also tried querying the external metrics API and I don't see the metrics there:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[{"name":"other-metric","singularName":"","namespaced":true,"kind":"ExternalMetricValueList","verbs":["get"]}]}

The keda-operator logs don't show any error and scale nicely from 1 to 0 and vice-versa

I found this Github issue but the solution was to add the permission to the worker node role, which I don't think is the right for this situation

Any idea where else I could look into?

[dcfranca]

Anyone??

[dcfranca]

Ok, the issue seems to be that we have kube-adapter-metrics already on the cluster and you can't have more than 1 external metrics server