While working with #11677 and #11637 I noticed a significant performance drop on my MoE tester. The drop was so significant that it could not have come from the SGL lists, so I tracked it down to 38c99d4
Environment
Node: 8× H100 + 8× NIC (HGX/DGX-class), NDR IB; one GPU per rank via CUDA_VISIBLE_DEVICES; mpirun, 8 ranks/node, inside a container
UCX: current master (includes 38c99d4 "UCP/PROTO: Distribute GPU-NIC matching across equal-bw lanes (#11547)")
NVML verified present/functional in the container: nvidia-smi -L and a direct dlopen("libnvidia-ml.so.1") + nvmlInit_v2 + nvmlDeviceGetCount_v2 probe both report 8 GPUs
Symptom Per-rank large-message GPU RMA bandwidth drops multi-× vs. before #11547. The application sends balanced data to all peers, yet UCX funnels all traffic through only 4 of 8 HCAs; the other 4 are ~200× idle.
Per-NIC port_xmit_data deltas, same workload, default (broken) vs. per-rank UCX_NET_DEVICES pinning:
HCA default (broken) pinned (fixed)
mlx5_0 231.5 GB 133.5 GB
mlx5_3 1.05 GB 133.5 GB
mlx5_4 249.0 GB 137.6 GB
mlx5_5 0.90 GB 134.5 GB
mlx5_6 224.4 GB 136.6 GB
mlx5_9 1.04 GB 128.4 GB
mlx5_10 237.5 GB 123.3 GB
mlx5_11 0.92 GB 127.4 GB
In the default case exactly 4 HCAs carry traffic (2 ranks each) and 4 are idle. Reverting 38c99d4, or pinning UCX_NET_DEVICES=:1 per rank, balances all 8 HCAs and restores full bandwidth.
Root cause ucp_proto_multi_find_max_avail_bw_lane() (src/ucp/proto/proto_multi.c) distributes the selected lane among equal-bandwidth NICs:
seed = req_sys_dev_ord % num_max_bw_devs; // GPU BDF ordinal
selected_sys_dev = sys_devs[seed]; // tied NICs, sorted by bus id
NVML enumeration (uct_cuda_enum_gpus) works here, so req_sys_dev_ord is a correct global GPU ordinal (0..7) — ordinals are not collapsing. Even so, the distribution maps the 8 ranks onto only 4 distinct NICs: the tied max-bw set resolves to num_max_bw_devs == 4 (the same 4 NICs for every GPU), so gpu_ord % 4 collides GPUs {0,4}, {1,5}, {2,6}, {3,7} onto shared HCAs and leaves the other 4 unused → 2:1 oversubscription (worse per-rank under contention).
Reproduction / evidence
nvidia-smi -L and NVML probe → 8 GPUs (rules out NVML/container GPU-hiding).
Per-NIC port_xmit_data deltas above (4 saturated / 4 idle by default; all 8 balanced when pinned).
Reverting 38c99d4 → all 8 HCAs used, full bandwidth.
UCX_NET_DEVICES=:1 per rank → all 8 HCAs used, full bandwidth.
Suggested fixes
- Guarantee distinct NIC assignment for distinct requesting devices when enough tied NICs exist; gpu_ord % num_max_bw_devs collides ranks whenever num_max_bw_devs < ranks_sharing_the_tied_set. Prefer the requesting GPU's topologically-affine NIC before falling back to round-robin.
- Investigate why the tied max-bw set resolves to a 4-NIC subset rather than all 8 on a symmetric topology — the distance→bandwidth fp8 quantization may be under-counting distinct devices.
- Provide a config opt-out to restore the previous per-rank affine selection.
While working with #11677 and #11637 I noticed a significant performance drop on my MoE tester. The drop was so significant that it could not have come from the SGL lists, so I tracked it down to 38c99d4
Environment
Node: 8× H100 + 8× NIC (HGX/DGX-class), NDR IB; one GPU per rank via CUDA_VISIBLE_DEVICES; mpirun, 8 ranks/node, inside a container
UCX: current master (includes 38c99d4 "UCP/PROTO: Distribute GPU-NIC matching across equal-bw lanes (#11547)")
NVML verified present/functional in the container: nvidia-smi -L and a direct dlopen("libnvidia-ml.so.1") + nvmlInit_v2 + nvmlDeviceGetCount_v2 probe both report 8 GPUs
Symptom Per-rank large-message GPU RMA bandwidth drops multi-× vs. before #11547. The application sends balanced data to all peers, yet UCX funnels all traffic through only 4 of 8 HCAs; the other 4 are ~200× idle.
Per-NIC port_xmit_data deltas, same workload, default (broken) vs. per-rank UCX_NET_DEVICES pinning:
HCA default (broken) pinned (fixed)
mlx5_0 231.5 GB 133.5 GB
mlx5_3 1.05 GB 133.5 GB
mlx5_4 249.0 GB 137.6 GB
mlx5_5 0.90 GB 134.5 GB
mlx5_6 224.4 GB 136.6 GB
mlx5_9 1.04 GB 128.4 GB
mlx5_10 237.5 GB 123.3 GB
mlx5_11 0.92 GB 127.4 GB
In the default case exactly 4 HCAs carry traffic (2 ranks each) and 4 are idle. Reverting 38c99d4, or pinning UCX_NET_DEVICES=:1 per rank, balances all 8 HCAs and restores full bandwidth.
Root cause ucp_proto_multi_find_max_avail_bw_lane() (src/ucp/proto/proto_multi.c) distributes the selected lane among equal-bandwidth NICs:
seed = req_sys_dev_ord % num_max_bw_devs; // GPU BDF ordinal
selected_sys_dev = sys_devs[seed]; // tied NICs, sorted by bus id
NVML enumeration (uct_cuda_enum_gpus) works here, so req_sys_dev_ord is a correct global GPU ordinal (0..7) — ordinals are not collapsing. Even so, the distribution maps the 8 ranks onto only 4 distinct NICs: the tied max-bw set resolves to num_max_bw_devs == 4 (the same 4 NICs for every GPU), so gpu_ord % 4 collides GPUs {0,4}, {1,5}, {2,6}, {3,7} onto shared HCAs and leaves the other 4 unused → 2:1 oversubscription (worse per-rank under contention).
Reproduction / evidence
nvidia-smi -L and NVML probe → 8 GPUs (rules out NVML/container GPU-hiding).
Per-NIC port_xmit_data deltas above (4 saturated / 4 idle by default; all 8 balanced when pinned).
Reverting 38c99d4 → all 8 HCAs used, full bandwidth.
UCX_NET_DEVICES=:1 per rank → all 8 HCAs used, full bandwidth.
Suggested fixes