Export NaNs in logits to scheduler_stats if output is corrupted

[vladmihailescu]

Signed-off-by: Vlad Mihailescu [email protected]

Summary:
Report nan in logits in scheduler_stats. This can be used later exported as Phrometeus counter but for now this is required so we can export it in our internal counter infra.

This counter is used to identify bad hosts or bad GPUs which cause NaNs in logits.

It's a common metric we expose.

Reviewed By: Adolfo-Karim

Differential Revision: D75423285

NO ENV VAR (NaN counting off):
I0618 22:24:27.611000 180881 reporter.py:195] Part 1: High-level performance metrics
Ran 404/404 requests in 56.56s
Success rate:        100.00%
QPS:                 7.14
Avg latency:         11.361s
Avg TTFT (client):   8245.31ms
P50 TTFT (client):   8262.13ms
P99 TTFT (client):   8283.14ms
Avg TTIT (client):   311.60ms
P50 TTIT (client):   323.78ms
P99 TTIT (client):   334.87ms
Avg TTFT (server):   6748.17ms
Avg TTIT (server):   357.83ms
Avg prefill len:     2587.54 tokens
P50 prefill len:     2587.00 tokens
P99 prefill len:     2596.00 tokens
Avg decode len:      10.00 tokens
P50 decode len:      10.00 tokens
P99 decode len:      10.00 tokens
I0618 22:22:13.811000 141492 reporter.py:195] Part 1: High-level performance metrics
Ran 404/404 requests in 56.30s
Success rate:        100.00%
QPS:                 7.18
Avg latency:         11.312s
Avg TTFT (client):   8209.60ms
P50 TTFT (client):   8214.03ms
P99 TTFT (client):   8273.95ms
Avg TTIT (client):   310.22ms
P50 TTIT (client):   322.10ms
P99 TTIT (client):   325.09ms
Avg TTFT (server):   6639.55ms
Avg TTIT (server):   355.36ms
Avg prefill len:     2587.49 tokens
P50 prefill len:     2587.00 tokens
P99 prefill len:     2595.00 tokens
Avg decode len:      10.00 tokens
P50 decode len:      10.00 tokens
P99 decode len:      10.00 tokens

WITH ENVVAR (NaN counting on):
I0618 22:44:09.103000 508486 reporter.py:195] Part 1: High-level performance metrics
Ran 404/404 requests in 56.18s
Success rate:        100.00%
QPS:                 7.19
Avg latency:         11.291s
Avg TTFT (client):   8193.95ms
P50 TTFT (client):   8208.52ms
P99 TTFT (client):   8232.08ms
Avg TTIT (client):   309.75ms
P50 TTIT (client):   321.92ms
P99 TTIT (client):   333.03ms
Avg TTFT (server):   8574.45ms
Avg TTIT (server):   355.91ms
Avg prefill len:     2587.90 tokens
P50 prefill len:     2588.00 tokens
P99 prefill len:     2596.00 tokens
Avg decode len:      10.00 tokens
P50 decode len:      10.00 tokens
P99 decode len:      10.00 tokens
I0618 22:47:21.294000 566871 reporter.py:195] Part 1: High-level performance metrics
Ran 404/404 requests in 57.23s
Success rate:        100.00%
QPS:                 7.06
Avg latency:         11.532s
Avg TTFT (client):   8365.60ms
P50 TTFT (client):   8229.34ms
P99 TTFT (client):   9156.80ms
Avg TTIT (client):   316.59ms
P50 TTIT (client):   322.71ms
P99 TTIT (client):   414.20ms
Avg TTFT (server):   6794.25ms
Avg TTIT (server):   358.08ms
Avg prefill len:     2587.78 tokens
P50 prefill len:     2587.00 tokens
P99 prefill len:     2597.00 tokens
Avg decode len:      10.00 tokens
P50 decode len:      10.00 tokens
P99 decode len:      10.00 tokens

And unit test

pytest -s -v tests/v1/worker/test_gpu_model_runner.py

tests/v1/worker/test_gpu_model_runner.py::test_get_nans_in_logits INFO 06-19 02:06:26 [config.py:1444] Using max model len 2048
WARNING 06-19 02:06:26 [config.py:4758] Current vLLM config is not set.
WARNING 06-19 02:06:26 [config.py:4758] Current vLLM config is not set.
WARNING 06-19 02:06:26 [topk_topp_sampler.py:59] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
PASSED

FIX #17123

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D75423285

[simon-mo]

Keep in mind that the logits needs to be serialized from model runner back to the scheduler via RPC/collectives. So doing the counting in model runner will be better.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D75423285

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D75423285

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D75423285

[markmc]

It doesn't seem like we actually need an exact count of nans - we just want a signal that corruption is spiking?

What happens the request when this happens? Is the request aborted, or ..?

Could we do something similar to the vllm:request_success_total metric - maybe add another FinishReason like CORRUPTED?

[markmc]

Some overlap with #18765 ... except I'm not sure this NaN case results in the request explicitly failing?

[vladmihailescu]

@markmc with NaNs the requests won't fail by default. With the existing behaviour the output of the requests will just be gibberish and that's why I'm not sure if we should add a new finish reason because NaNs will not forcefully fail the request.

Internally we observe 2 behaviours in which NaNs appear:

1. Bad GPU - this causes all the requests landing on that GPU to have NaNs. We fail warmup if this happens during warmup (so the container restarts and retries warmup) but if it happens while the task is healthy, the container won't stop because there can be random flaky NaNs (see point 2).
2. Flaky - very rarely happens that a request will have NaNs in logits even though it was processed on a healthy GPUs.

I am trying to add now the per request NaNs and maybe have a bool per request telling if it's corrupted or not, but not sure if we should make it a finish reason.

[houseroad]

Looks good to me. Thanks for adding this metrics.

[vladmihailescu]

V1 test timed out. Rebasing after the fix (#19872)

[yeqcharlotte]

is logits optionally None here or not?

[yeqcharlotte]

nit: you got 2 if logits is not none check here and one in the loop. for better readability it might make sense to do the following:

if logits is None:
    return {req_id: 0 for ...}
num_nans_for_index = logits.isnan().sum(dim=-1).cpu().numpy()    
# count nans

[yeqcharlotte]

thanks for adding this! leaving some small nits only.

[vladmihailescu]

Addressed nits