What happened + What you expected to happen
We are running Kuberay-managed Ray Serve version 2.54.0 and have k8s health checks with 5s frequency and 2s expiry. We have also configured Prometheus scrapes to occur every 60s.
We noticed that after receiving bursts of traffic, some of our pods would begin randomly failing their health checks. This condition persisted even after the traffic went away.
We noticed that the health checks were not failing completely randomly, but roughly once every 60s (our Prometheus metric scrap interval), so we suspected that they were related somehow. We checked the metrics and noticed that many individual lines were being shipped. These had the most lines:
ray_serve_request_router_fulfillment_time_ms_bucket
ray_serve_request_router_fulfillment_time_ms_sum
ray_serve_request_router_fulfillment_time_ms_count
ray_serve_num_router_requests_total
ray_serve_handle_request_counter_total
A Speedscope trace of the Dashboard Agent confirmed our suspicions. In the trace below, a successful healthz check was performed at t=15s. The next check is scheduled at t=20s'. However, between 17s and 24s, metrics were being generated. As a result, the health check that was supposed to happen at t=20s was delayed to after t=24s (visible as a thin purple sliver), past the 2s timeout.
Related issues:
Versions / Dependencies
- Ray Serve version 2.54.0
- Kuberay 1.4
- RAY_enable_open_telemetry=true
- health checks with frequency of 5s and expiry of 2s
- Prometheus metrics scrape interval of 60s
Reproduction script
This was a production issue that is rather difficult to reproduce from scratch. The trigger condition is when the Ray cluster receives large volumes of bursty traffic, which triggered frequent pod scale-up/down. The traffic causes metric counts to balloon, as it now tracks all the created/deleted WorkerIds. That in turn increases the metric scrape time.
Issue Severity
Medium: It is a significant difficulty but I can work around it.
What happened + What you expected to happen
We are running Kuberay-managed Ray Serve version 2.54.0 and have k8s health checks with 5s frequency and 2s expiry. We have also configured Prometheus scrapes to occur every 60s.
We noticed that after receiving bursts of traffic, some of our pods would begin randomly failing their health checks. This condition persisted even after the traffic went away.
We noticed that the health checks were not failing completely randomly, but roughly once every 60s (our Prometheus metric scrap interval), so we suspected that they were related somehow. We checked the metrics and noticed that many individual lines were being shipped. These had the most lines:
A Speedscope trace of the Dashboard Agent confirmed our suspicions. In the trace below, a successful healthz check was performed at t=15s. The next check is scheduled at t=20s'. However, between 17s and 24s, metrics were being generated. As a result, the health check that was supposed to happen at t=20s was delayed to after t=24s (visible as a thin purple sliver), past the 2s timeout.
Related issues:
Versions / Dependencies
Reproduction script
This was a production issue that is rather difficult to reproduce from scratch. The trigger condition is when the Ray cluster receives large volumes of bursty traffic, which triggered frequent pod scale-up/down. The traffic causes metric counts to balloon, as it now tracks all the created/deleted WorkerIds. That in turn increases the metric scrape time.
Issue Severity
Medium: It is a significant difficulty but I can work around it.