Add an option to disable GPU metric polling

[pawelbeza]

Description

Description

Ray’s dashboard reporter periodically polls NVML for GPU metrics. In some
environments, these calls can contend with other GPU operations and contribute
to kernel lockups or panics.

Reducing the polling frequency only mitigates the problem. Deployments using
external GPU observability need a way to disable NVML polling completely.

Proposed solution

Add a default-enabled RayConfig option:

ray.init(
    _system_config={"enable_gpu_metrics_collection": False}

Use case

No response

[Kunchd]

Hi @pawelbeza, thanks for the issue! I'll take a look at the PR. In the meantime, do you have a repro script that can show the issue mentioned?

[elnikety]

Hi @pawelbeza, thanks for the issue! I'll take a look at the PR. In the meantime, do you have a repro script that can show the issue mentioned?

Thanks @Kunchd -- The contention is hard to reproduce outside production systems at scale. Overall, I think a knob to allow a system administrator to tune which system software can / may report on GPU status is useful to have regardless.