ClusterLoad: Sometimes the connection with Prometheus gets closed

[ebenahar]

In ClusterLoad (IO in BG) we constantly sample different metrics using Prometheus.

It looks like an issue with the connection with Prometheus, that started below (seen here).

[2025-01-12T09:34:04.153Z] 04:33:53 - Thread-3 - ocs_ci.utility.utils - ESC[1mESC[31mERRORESC[0m - Exception raised during iteration: HTTPSConnectionPool(host='prometheus-k8s-openshift-monitoring.apps.j006aikv23fc333t1.qe.rh-ocs.com', port=443): Max retries exceeded with url: /api/v1/query?query=cluster%3Aceph_disk_latency%3Ajoin_ceph_node_disk_irate1m&time=1736674429.285991 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fea2bcb7160>: Failed to establish a new connection: [Errno -2] Name or service not known'))