Skip to content

[Bug]: vllm serving deepseek-r1-0528 model on AWS EFA with 8*H200 #28087

New issue
New issue
Open
Open
[Bug]: vllm serving deepseek-r1-0528 model on AWS EFA with 8*H200#28087
Labels
bugSomething isn't working
@CalebZ9909

Description

@CalebZ9909
CalebZ9909
opened on Nov 5, 2025

Your current environment

<details>

The output of python collect_env.py 
Your output of `python collect_env.py` here

🐛 Describe the bug

Your current environment

Collecting environment information...

    System Info

==============================
OS : Ubuntu 22.04.5 LTS (x86_64)
GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Clang version : 14.0.0-1ubuntu1.1
CMake version : version 4.1.0
Libc version : glibc-2.35

==============================
PyTorch Info

PyTorch version : 2.8.0+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A

==============================
Python Environment

Python version : 3.13.5 | packaged by Anaconda, Inc. | (main, Jun 12 2025, 16:09:02) [GCC 11.2.0] (64-bit runtime)
Python platform : Linux-6.8.0-1036-aws-x86_64-with-glibc2.35

==============================
CUDA / GPU Info

Is CUDA available : True
CUDA runtime version : 12.8.93
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration :
GPU 0: NVIDIA H200
GPU 1: NVIDIA H200
GPU 2: NVIDIA H200
GPU 3: NVIDIA H200
GPU 4: NVIDIA H200
GPU 5: NVIDIA H200
GPU 6: NVIDIA H200
GPU 7: NVIDIA H200

Nvidia driver version : 570.172.08
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True

==============================
CPU Info

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 96
On-line CPU(s) list: 0-95
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8488C
CPU family: 6
Model: 143
Thread(s) per core: 1
Core(s) per socket: 48
Socket(s): 2
Stepping: 8
BogoMIPS: 4800.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 4.5 MiB (96 instances)
L1i cache: 3 MiB (96 instances)
L2 cache: 192 MiB (96 instances)
L3 cache: 210 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-47
NUMA node1 CPU(s): 48-95
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

==============================
Versions of relevant libraries

[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.57.0
[pip3] triton==3.4.0
[conda] numpy 2.2.6 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi
[conda] nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi
[conda] nvidia-ml-py 13.580.82 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.27.3 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi
[conda] pyzmq 27.1.0 pypi_0 pypi
[conda] torch 2.8.0 pypi_0 pypi
[conda] torchaudio 2.8.0 pypi_0 pypi
[conda] torchvision 0.23.0 pypi_0 pypi
[conda] transformers 4.57.0 pypi_0 pypi
[conda] triton 3.4.0 pypi_0 pypi

==============================
vLLM Info

ROCM Version : Could not collect
vLLM Version : 0.11.0rc2.dev332+gbb6d8c21f (git sha: bb6d8c2)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 0-47 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 0-47 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 0-47 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 0-47 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 48-95 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 48-95 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 48-95 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X 48-95 1 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

==============================
Environment Variables

PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I am running VLLM on the deepseek-ai/DeepSeek-R1-0528 model on 2 AWS EFA 8*H200 nodes. but I hit the following error:
on node 1:

INFO 01-05 01:44:18 engine.py:184] Initializing an LLM engine (v0.11.0) with
 config: ...
 INFO 01-05 01:44:22 model_runner.py:1172] Capturing cudagraphs for decoding. 
 This may lead to unexpected consequences if the model is not static. To run 
 the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in
  the CLI.
 INFO 01-05 01:44:22 model_runner.py:1176] If out-of-memory error occurs 
 during cudagraph capture, consider decreasing `gpu_memory_utilization` or 
 switching to eager mode. You can also reduce the `max_num_seqs` as needed to 
 decrease memory usage.

 [DeepGEMM JIT] Looking for deep_gemm cache in
 /fsx/ubuntu/.cache/vllm/deep_gemm/cache
 [DeepGEMM JIT] Compiling new kernel...
 [DeepGEMM JIT] Config: 
 m_grouped_fp8_gemm_contiguous_1d2d.e9fb3acaaf9356c88974ac87fd24c594
 Running NVCC command: /usr/local/cuda-12.8/bin/nvcc --std=c++17 -gencode
 arch=compute_90,code=sm_90 -O3 -Xcompiler -O3 -Xcompiler -fPIC --shared -o
 /fsx/ubuntu/.cache/vllm/deep_gemm/cache/kernel.sm90_m_grouped_fp8_gemm_contig
 uous_1d2d.e9fb3acaaf9356c88974ac87fd24c594/kernel.so 
 /fsx/ubuntu/.cache/vllm/deep_gemm/cache/kernel.sm90_m_grouped_fp8_gemm_contig
 uous_1d2d.e9fb3acaaf9356c88974ac87fd24c594/kernel.cu

on node 2:

INFO 01-05 01:44:17 engine.py:184] Initializing an LLM engine (v0.11.0) with
  config: ...
  INFO 01-05 01:44:21 model_runner.py:1172] Capturing cudagraphs for decoding. 
  This may lead to unexpected consequences if the model is not static...

  [DeepGEMM JIT] Looking for deep_gemm cache in
  /fsx/ubuntu/.cache/vllm/deep_gemm/cache
  [DeepGEMM JIT] Compiling new kernel...
  [DeepGEMM JIT] Config: 
  m_grouped_fp8_gemm_contiguous_1d2d.e9fb3acaaf9356c88974ac87fd24c594

  Running NVCC command: /usr/local/cuda-12.8/bin/nvcc --std=c++17 -gencode
  arch=compute_90,code=sm_90 -O3 -Xcompiler -O3 -Xcompiler -fPIC --shared -o
  /fsx/ubuntu/.cache/vllm/deep_gemm/cache/kernel.sm90_m_grouped_fp8_gemm_contig
  uous_1d2d.e9fb3acaaf9356c88974ac87fd24c594/kernel.so 
  /fsx/ubuntu/.cache/vllm/deep_gemm/cache/kernel.sm90_m_grouped_fp8_gemm_contig
  uous_1d2d.e9fb3acaaf9356c88974ac87fd24c594/kernel.cu

  Traceback (most recent call last):
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/engine/core.py", line 303, in 
  run_busy_loop
      await self._run_busy_loop()
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/engine/core.py", line 334, in 
  _run_busy_loop
      await asyncio.sleep(0)
    File
  "/fsx/ubuntu/miniconda3/envs/vllm_py311/lib/python3.11/asyncio/tasks.py",
  line 649, in sleep
      return await future
  asyncio.exceptions.CancelledError

  During handling of the above exception, another exception occurred:
   Traceback (most recent call last):
    File "/fsx/ubuntu/miniconda3/envs/vllm_py311/lib/python3.11/site-packages/v
  llm/v1/engine/async_llm.py", line 121, in engine_step
      await self.engine.step_async()
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/engine/llm_engine.py", line 119,
  in step_async
      await self.engine_core.step_async(scheduler_output)
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/engine/core.py", line 196, in 
  step_async
      self.core_task = task
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/engine/core.py", line 311, in
  run_busy_loop
      await self._wait_for_tasks()
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/engine/core.py", line 283, in
  _wait_for_tasks
      raise result
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/engine/core.py", line 260, in
  _run_worker_task
      result = await result
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/worker/gpu_worker.py", line 90,
  in execute_model_async
      output = await self.model_runner.execute_model_async(
    File "/fsx/ubuntu/uccl_yihan/vllm/vllm/v1/worker/gpu_model_runner.py", line
   408, in execute_model_async
      return await self.model.execute_model_async(execute_model_input,
    File
  "/fsx/ubuntu/uccl_yihan/vllm/vllm/model_executor/models/deepseek_r1.py", line
   659, in execute_model_async
      hidden_states = self.model(
    File "/fsx/ubuntu/miniconda3/envs/vllm_py311/lib/python3.11/site-packages/t
  orch/nn/modules/module.py", line 1909, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
    File "/fsx/ubuntu/miniconda3/envs/vllm_py311/lib/python3.11/site-packages/t
  orch/nn/modules/module.py", line 1918, in _call_impl
      return forward_call(*args, **kwargs)
    File
  "/fsx/ubuntu/uccl_yihan/vllm/vllm/model_executor/models/deepseek_r1.py", line
   490, in forward
      hidden_states = layer(
    File "/fsx/ubuntu/miniconda3/envs/vllm_py311/lib/python3.11/site-packages/t
  orch/nn/modules/module.py", line 1909, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
    File "/fsx/ubuntu/miniconda3/envs/vllm_py311/lib/python3.11/site-packages/t
  orch/nn/modules/module.py", line 1918, in _call_impl
      return forward_call(*args, **kwargs)
    File
  "/fsx/ubuntu/uccl_yihan/vllm/vllm/model_executor/models/deepseek_r1.py", line
   382, in forward
      hidden_states = self.mlp(hidden_states)
    File "/fsx/ubuntu/miniconda3/envs/vllm_py311/lib/python3.11/site-packages/t
  orch/nn/modules/module.py", line 1909, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
    File "/fsx/ubuntu/miniconda3/envs/vllm_py311/lib/python3.11/site-packages/t
  orch/nn/modules/module.py", line 1918, in _call_impl
      return forward_call(*args, **kwargs)
    File
  "/fsx/ubuntu/uccl_yihan/vllm/vllm/model_executor/models/deepseek_r1.py", line
   229, in forward
      return self.forward_v2(x)
    File
  "/fsx/ubuntu/uccl_yihan/vllm/vllm/model_executor/models/deepseek_r1.py", line
   245, in forward_v2
      x_grouped = grouped_gemm_v2(
    File
  "/fsx/ubuntu/uccl_yihan/DeepGEMM/deep_gemm/primitives/grouped_gemm.py", line
  218, in grouped_gemm_v2
      return impl.run(
    File
  "/fsx/ubuntu/uccl_yihan/DeepGEMM/deep_gemm/primitives/grouped_gemm.py", line
  137, in run
      y = m_grouped_fp8_gemm_nt_contiguous(
    File "/fsx/ubuntu/uccl_yihan/DeepGEMM/deep_gemm/jit_kernels/impls/grouped_g
  emm.py", line 198, in m_grouped_fp8_gemm_nt_contiguous
      handle = load_kernel(
    File "/fsx/ubuntu/uccl_yihan/DeepGEMM/deep_gemm/jit/handle.py", line 83, in
   load_kernel
      raise RuntimeError(f'CUDA driver error
  (csrc/apis/../jit_kernels/impls/../../jit/handle.hpp:83): {result.value}
  ({cuda_error_enum_to_str(result)}, {cuda_error_enum_to_message(result)})')
  RuntimeError: CUDA driver error
  (csrc/apis/../jit_kernels/impls/../../jit/handle.hpp:83): 301
  (CUDA_ERROR_FILE_NOT_FOUND, file not found)

It seems like:
16 processes (2 nodes × 8 GPU workers per node) started simultaneously and all needed to compile the same DeepGEMM CUDA kernel.
All 16 processes simultaneously:

  1. Detected the kernel was missing from cache
  2. Started NVCC compilation, writing to the same file:
    /fsx/ubuntu/.cache/vllm/deep_gemm/cache/kernel.sm90_m_grouped_fp8_gemm_contig
    uous_1d2d.e9fb3acaaf9356c88974ac87fd24c594/kernel.so
  3. Overwrote each other's compilation output

Result:

  • When Node2's 8 processes tried to load the kernel, the file was corrupted
    or incomplete
  • Error: RuntimeError: CUDA driver error 301 (CUDA_ERROR_FILE_NOT_FOUND)
  • Node1 detected Node2 process crashes, NCCL connections failed

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    DeepSeek V3/R1

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions