Rook v1.13.1 / Ceph 17.2.7 - New Cluster Utilization Dashboard has no metrics

[reefland]

Metrics were working on Ceph 17.2.6, upgraded to 17.2.7 and the new Cluster Utilization dashboard shows no metrics:

• I'm aware this new dashboard can be turned off to use the previous one. But I'm looking to get the new dashboard working.
• I'm aware that Ceph 18.2.1 seems to be the new Helm default, I'm holding out on 17.2 for a bit more.

Grafana Integration itself still seems fine, such as performance details of a host:

Prometheus Scrapes are working:

I don't see any error message in the MGR, filtered on keyword dashboard:

debug 2024-01-08T18:50:44.320+0000 7fb5e6972700  0 [dashboard INFO request] [192.168.10.215:55055] [GET] [200] [1.820s] [admin] [268.0B] /ceph-dashboard/api/host
debug 2024-01-08T18:50:46.280+0000 7fb5e496e700  0 [dashboard INFO request] [192.168.10.215:55055] [GET] [200] [0.017s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:50:46.288+0000 7fb5e7173700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.019s] [admin] [1.5K] /ceph-dashboard/api/summary
debug 2024-01-08T18:50:47.496+0000 7fb5e296a700  0 [dashboard INFO orchestrator] is orchestrator available: True,
debug 2024-01-08T18:50:47.500+0000 7fb5e296a700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.230s] [admin] [529.0B] /ceph-dashboard/ui-api/orchestrator/status
debug 2024-01-08T18:50:47.712+0000 7fb5e5970700  0 [dashboard INFO orchestrator] is orchestrator available: True,
debug 2024-01-08T18:50:48.176+0000 7fb5e5970700  0 [dashboard INFO orchestrator] is orchestrator available: True,
debug 2024-01-08T18:50:48.376+0000 7fb5e5970700  0 [dashboard INFO orchestrator] is orchestrator available: True,
debug 2024-01-08T18:50:48.576+0000 7fb5e5970700  0 [dashboard INFO orchestrator] is orchestrator available: True,
debug 2024-01-08T18:50:48.776+0000 7fb5e5970700  0 [dashboard INFO orchestrator] is orchestrator available: True,
debug 2024-01-08T18:50:48.988+0000 7fb5e5970700  0 [dashboard INFO orchestrator] is orchestrator available: True,
debug 2024-01-08T18:50:48.996+0000 7fb5e5970700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [1.491s] [admin] [268.0B] /ceph-dashboard/api/host
debug 2024-01-08T18:50:51.460+0000 7fb5e416d700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.005s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:50:57.464+0000 7fb5e396c700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.003s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:02.460+0000 7fb5e516f700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.004s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:07.460+0000 7fb5e6171700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.004s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:12.456+0000 7fb5e316b700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.003s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:17.456+0000 7fb5e6972700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.003s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:22.460+0000 7fb5e496e700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.004s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:27.476+0000 7fb5e7173700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.006s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:32.476+0000 7fb5e296a700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.003s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:37.476+0000 7fb5e5970700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.003s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:42.480+0000 7fb5e416d700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.005s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:47.476+0000 7fb5e396c700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.004s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:52.476+0000 7fb5e516f700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.004s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications
debug 2024-01-08T18:51:57.500+0000 7fb5e6171700  0 [dashboard INFO request] [192.168.10.215:12048] [GET] [200] [0.003s] [admin] [22.0B] /ceph-dashboard/api/prometheus/notifications

Suggestions welcomed.

[travisn]

@avanthakkar Thoughts on why the metrics wouldn't be there in v17.2.7? This should be affected by the exporter since that's new in v18.

[reefland]

Bumped to Rook v1.13.2 / Ceph 17.2.7 still no dashboard metrics. I'll bump Ceph to 18.2.1 shortly see if that helps.

[reefland]

I upgraded to Ceph 18.2.1, nothing eventful happened. All looks good:

rook-ceph-crashcollector-k3s01          req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-crashcollector-k3s02          req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-crashcollector-k3s03          req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-crashcollector-k3s04          req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-crashcollector-k3s05          req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-crashcollector-k3s06          req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-exporter-k3s01        req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-exporter-k3s02        req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-exporter-k3s03        req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-exporter-k3s04        req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-exporter-k3s05        req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-exporter-k3s06        req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-mds-ceph-filesystem-a         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-mds-ceph-filesystem-b         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-mgr-a         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-mgr-b         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-mon-w         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-mon-y         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-mon-z         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-0         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-1         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-2         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-3         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-4         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-5         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-6         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-7         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-osd-8         req/upd/avl: 1/1/1      rook-version=v1.13.2    ceph-version=18.2.1-0
rook-ceph-rgw-ceph-objectstore-a        req/upd/avl: 2/2/2      rook-version=v1.13.2    ceph-version=18.2.1-0

  cluster:
    id:     cb82340a-2eaf-4597-b83e-cc0e62a9d019
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum w,y,z (age 16m)
    mgr: b(active, since 16m), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 9 osds: 9 up (since 11m), 9 in (since 2w)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   12 pools, 193 pgs
    objects: 89.17k objects, 231 GiB
    usage:   713 GiB used, 6.2 TiB / 6.9 TiB avail
    pgs:     193 active+clean
 
  io:
    client:   1.2 KiB/s rd, 87 KiB/s wr, 2 op/s rd, 8 op/s wr

ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)

Prometheus shows the rook-ceph-exporter targets are up:

However, the rook-ceph-mgr is not reachable:

That is pointing to the active MGR:

$ k get pods -n rook-ceph -o wide -l mgr_role=active
NAME                               READY   STATUS    RESTARTS   AGE   IP               NODE    NOMINATED NODE   READINESS GATES
rook-ceph-mgr-b-744bfb9d85-bpsmx   3/3     Running   0          49m   192.168.10.216   k3s02   <none>           <none>

I'm getting a Module not found error:

debug 2024-01-17T01:42:38.703+0000 7f1f32199700 -1 no module 'rook'
debug 2024-01-17T01:42:38.703+0000 7f1f32199700 -1 mgr load Failed to construct class in 'prometheus'
debug 2024-01-17T01:42:38.707+0000 7f1f096b5700  0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7f1f1579f5f8>, <progress.module.GhostEvent object at 0x7f1f1579f5c0>, <progress.module.GhostEvent object at 0x7f1f1579f588>, <progress.module.GhostEvent object at 0x7f1f1579fb38>, <progress.module.GhostEvent object at 0x7f1f1579fac8>, <progress.module.GhostEvent object at 0x7f1f1579fb00>, <progress.module.GhostEvent object at 0x7f1f1579fa90>, <progress.module.GhostEvent object at 0x7f1f1579fa58>, <progress.module.GhostEvent object at 0x7f1f1579fa20>, <progress.module.GhostEvent object at 0x7f1f1579f9e8>, <progress.module.GhostEvent object at 0x7f1f1579f9b0>, <progress.module.GhostEvent object at 0x7f1f1579f978>, <progress.module.GhostEvent object at 0x7f1f1579f940>, <progress.module.GhostEvent object at 0x7f1f1579f908>, <progress.module.GhostEvent object at 0x7f1f1579f8d0>, <progress.module.GhostEvent object at 0x7f1f1579f898>, <progress.module.GhostEvent object at 0x7f1f1579fe10>, <progress.module.GhostEvent object at 0x7f1f1579fda0>, <progress.module.GhostEvent object at 0x7f1f1579fdd8>, <progress.module.GhostEvent object at 0x7f1f1579fd68>, <progress.module.GhostEvent object at 0x7f1f1579fd30>, <progress.module.GhostEvent object at 0x7f1f1579fcf8>, <progress.module.GhostEvent object at 0x7f1f1579fcc0>, <progress.module.GhostEvent object at 0x7f1f1579fc88>, <progress.module.GhostEvent object at 0x7f1f1579fc50>, <progress.module.GhostEvent object at 0x7f1f1579fc18>, <progress.module.GhostEvent object at 0x7f1f1579fbe0>, <progress.module.GhostEvent object at 0x7f1f1579fba8>, <progress.module.GhostEvent object at 0x7f1f1579fb70>, <progress.module.GhostEvent object at 0x7f1f1579ffd0>, <progress.module.GhostEvent object at 0x7f1f1579ff98>, <progress.module.GhostEvent object at 0x7f1f1579ff60>, <progress.module.GhostEvent object at 0x7f1f1579ff28>, <progress.module.GhostEvent object at 0x7f1f1579fef0>, <progress.module.GhostEvent object at 0x7f1f1579feb8>, <progress.module.GhostEvent object at 0x7f1f1579fe80>, <progress.module.GhostEvent object at 0x7f1f1579fe48>, <progress.module.GhostEvent object at 0x7f1f1579f550>, <progress.module.GhostEvent object at 0x7f1f1579f518>, <progress.module.GhostEvent object at 0x7f1f1579f4e0>, <progress.module.GhostEvent object at 0x7f1f1579f4a8>, <progress.module.GhostEvent object at 0x7f1f1579f470>, <progress.module.GhostEvent object at 0x7f1f157c6048>, <progress.module.GhostEvent object at 0x7f1f157c6080>, <progress.module.GhostEvent object at 0x7f1f157c60b8>, <progress.module.GhostEvent object at 0x7f1f157c60f0>, <progress.module.GhostEvent object at 0x7f1f157c6128>, <progress.module.GhostEvent object at 0x7f1f157c6160>, <progress.module.GhostEvent object at 0x7f1f157c6198>, <progress.module.GhostEvent object at 0x7f1f157c61d0>] historic events
debug 2024-01-17T01:42:38.707+0000 7f1f0a6b7700  0 [pg_autoscaler INFO root] _maybe_adjust
debug 2024-01-17T01:42:38.707+0000 7f1f096b5700  0 [progress INFO root] Loaded OSDMap, ready.
debug 2024-01-17T01:42:38.703+0000 7f1f32199700 -1 mgr load Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1657, in _oremote
    return mgr.remote(o, meth, *args, **kwargs)
  File "/usr/share/ceph/mgr/mgr_module.py", line 2228, in remote
    args, kwargs)
ImportError: Module not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/prometheus/module.py", line 649, in __init__
    self.modify_instance_id = self.get_orch_status() and self.get_module_option(
  File "/usr/share/ceph/mgr/prometheus/module.py", line 869, in get_orch_status
    return self.available()[0]
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1586, in inner
    completion = self._oremote(method_name, args, kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1661, in _oremote
    f_set = self.get_feature_set()
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1586, in inner
    completion = self._oremote(method_name, args, kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1657, in _oremote
    return mgr.remote(o, meth, *args, **kwargs)
  File "/usr/share/ceph/mgr/mgr_module.py", line 2228, in remote
    args, kwargs)
ImportError: Module not found

debug 2024-01-17T01:42:38.735+0000 7f1f32199700 -1 mgr operator() Failed to run module in active mode ('prometheus')

I've restart the MGR and Operator, message remains.

[reefland]

$ ceph mgr module ls
MODULE                              
balancer              on (always on)
crash                 on (always on)
devicehealth          on (always on)
orchestrator          on (always on)
pg_autoscaler         on (always on)
progress              on (always on)
rbd_support           on (always on)
status                on (always on)
telemetry             on (always on)
volumes               on (always on)
dashboard             on            
iostat                on            
nfs                   on            
prometheus            on            
restful               on            
rook                  on            
alerts                -             
cephadm               -             
diskprediction_local  -             
influx                -             
insights              -             
k8sevents             -             
localpool             -             
mds_autoscaler        -             
mirroring             -             
osd_perf_query        -             
osd_support           -             
rgw                   -             
selftest              -             
snap_schedule         -             
stats                 -             
telegraf              -             
test_orchestrator     -             
zabbix                -             

[reefland]

Looks like I'm facing #13527

[rkachach]

Indeed, once you have upgraded to v18.2.1 you hit the BUG #13527. Although not sure what was the original issue preventing metrics from not showing up in v17.2.7.