Dashboard with external prometheus

[arichard42]

Hi,

I am deploying rook-ceph-cluster using an external kube-prometheus-stack.

The relevant values I use to deploy the operator are :

monitoring:
  enabled: true

And the cluster relevant values :

monitoring:
  enabled: true
  createPrometheusRules: true
  rulesNamespaceOverride: monitoring
  prometheusRule:
    labels:
      release: prometheus
cephObjectStores:
  - name: ceph-objectstore
    spec:
      gateway:
        port: 80
        instances: 2

I am currently experiencing two well identified problems with dashboard performance graph :

1. Disk utilization

Some graphs are showing empty are N/A indication for some disk io statistics.

For example one Cluster/Hosts/Overall Performance, the AVG Disk Utilization frame show N/A.
Stats about latencies on Cluster/OSDs/Overall Performance are also empty.

Looking on the prometheus/grafana side, it appears this is because prometheus have replaced the instance label with a host:port indication, copying the original instance label in an exported_instance label.

This is because the relevant serviceMonitor/rook-ceph/rook-ceph-mgr generated by rook operator do not have the "honor_labels: true" indication.

Editing manually the serviceMonitor/rook-ceph/rook-ceph-mgr resource solves the problem, but this is not a long term solution as this resource is generated internaly by the operator at the next operator or cluster update.

2. Object store with two instances

When I deploy the object store with two instances, the two instances are using the same name and id :

ceph orch ps --daemon_type=rgw
NAME                  HOST                 PORTS  STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION    IMAGE ID      
rgw.ceph-objectstore  kub4.equation.intra         running (3w)     0s ago   3w        -        -  <unknown>  8c1697a0a924  
rgw.ceph-objectstore  kub5.equation.intra         running (6w)     0s ago   6w        -        -  <unknown>  798f1b1e71ca  

and most of the object gateway dashboard shows no data with an error "found duplicate series for the match group".

Looking at a cephadm (non-rook) deployed cluter, the instances get different ids :

ceph orch ps --daemon_type=rgw
NAME                       HOST                      PORTS  STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  
rgw.test.linuxagr2.gxvxyf  linuxagr2.equation.intra  *:80   running (63m)     2m ago  63m    85.9M        -  18.2.2   3183778aa140  bf83fe3c2c27  
rgw.test.linuxagr.zxbvhr   linuxagr.equation.intra   *:80   running (63m)     2m ago  63m    90.5M        -  18.2.2   3183778aa140  307f520d3dd5  

The only solution so far is to have a single gw instance.

[BlaineEXE]

Regarding the RGW instances in the dashboard, I think @rkachach has done some investigation into that.

Regarding the monitoring labels, you can provide additional labels on the mgr using this mechanism: https://rook.io/docs/rook/latest-release/CRDs/Cluster/ceph-cluster-crd/#annotations-and-labels

[BlaineEXE]

I also don't fully understand what you mean here:

Looking on the prometheus/grafana side, it appears this is because prometheus have replaced the instance label with a host:port indication, copying the original instance label in an exported_instance label.

If you could show us in an example what you mean, it could help us get more clear about what you're trying to explain and how it is proving to be a problem. It seems that another means of addressing this may be to alter the metrics to not use host:port.

[rkachach]

As @BlaineEXE explained I was looking into the no-unique RGW instance id issue for a while. It's a known issue which is being tracked by #13674, unfortunately at this moment we don't have a definitive solution yet. As for the point nº 1 if you are sure that's a rook issue I'd recommend opening a BUG report and providing as much detailed information as you can.