What is required api rights for prometheus plugin?

[feym78]

Hi. I'm running in-cluster installation of headlamp 0.23.1.
When I use cluster-admin or edit cluster role then prometheus chart is working fine for pods, but if I use view role or custom set of privileges, then chart is not showing with error "Error fetching prometheus Info"
I managed to get understanding that plugin is trying to search all namespaces to see if there is prometheus pod, so I added previleges to get pods in all namespaces, error message disappear, but chart still not showing, in browser console I see error "GET generated url with prometheus pod 403 (Forbidden)".

I can't understand which api right is missing, I tried to grant all api resources in "" group, "apps" group and etc with full scope ("get", "list", "watch", "create", "update", "patch", "delete") clusterwide, but still no result.

My idea is that I don't want to give full access to the cluster for team or even edit role clusterwide and give as few as possible api scope for token, but still provide necessary metrics and tools.

[joaquimrocha]

@yolossn Can you reply this one?

[yolossn]

Hey @feym78 The plugin identifies the prometheus pod using the app.kubernetes.io/name=prometheus label, can you check if your prometheus pod has this label?

kubectl get pods -l app.kubernetes.io/name=prometheus --all-namespace

We are working on making the prometheus url configurable so that the user can configure the prometheus endpoint to fetch the metrics from.

[feym78]

@yolossn Hi, thx for reply
Yes, in my installation prometheus pod has this label (your query works, but with -A instead of "--all-namespace").
Thing is that plugin works with admin or edit cluster role, but I can't figure out which api rights necessary with custom set of rights.
I'm trying to setup "read-only" role in k8s or with limited edit rights in cluster. For testing purposes I tried to grant all api resources in "" group, "apps" group and etc with full scope ("get", "list", "watch", "create", "update", "patch", "delete") clusterwide, but still no result.

[feym78]

@yolossn Hi! Any thoughts on this?

[yolossn]

Hey sorry, I missed this message. I think the reason why the read-only setup is not working is that the plugin creates a proxy to the prometheus plugin for fetching the metrics data. Can you share the output of this command with the "read-only" grant.

kubectl auth can-i create pods/proxy
kubectl auth can-i create services/proxy

[feym78]

Hey sorry, I missed this message. I think the reason why the read-only setup is not working is that the plugin creates a proxy to the prometheus plugin for fetching the metrics data. Can you share the output of this command with the "read-only" grant.

kubectl auth can-i create pods/proxy kubectl auth can-i create services/proxy

Strange thing - with admin role I get "yes" to this commands, but if I try them even with full scope for cluster role as:

• apiGroups: [""]
  resources: ["pods/proxy", "services/proxy"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  I still getting "no".
  BUT prometheus chart starts to work with this scope:
• apiGroups: [""]
  resources: ["pods/proxy", "services/proxy"]
  verbs: ["get"]
  even more, it still works with this:
• apiGroups: [""]
  resources: ["pods/proxy"]
  verbs: ["get"]

So in order to get prometheus plugin working you need to grant SA get/list to all namespaces and pods (so plugin could find prometheus pod) and scope above for pods/proxy.
Thanks a lot for your help!

As offtop: now it would be nice to make chart time range customizable, not hardcoded 10 minutes :)