`uct_iface_open` failure on unusable RDMA device aborts entire worker instead of skipping

[evacchi]

Describe the bug

In a privileged Kubernetes pod, UCX discovers host RDMA management sub-functions (mlx5_0–mlx5_8) that cannot serve RC verbs. uct_iface_open(rc_verbs/mlx5_5:1) fails with EADDRNOTAVAIL, which aborts the entire UCX worker creation. The 8 usable IB data ports (ibp0–ibp7) are never tried.

Non-privileged pods work fine: the k8s RDMA device plugin only injects the usable ibp* devices into /sys/class/infiniband/.

The expected behavior would be not to enumerate devices that would fail uct_iface_open. For instance, devices with all-zero GID tables or whose PCI function cannot serve RC verbs should be excluded during discovery. If pre-filtering is not feasible, UCX should skip failing devices and continue with the remaining usable ones (as #9475 does for TCP interfaces). An explicit flag similar to UCX_TCP_BRIDGE_ENABLE (#9475) might be also acceptable.

Steps to Reproduce

1. Run a privileged pod on a node with both IB data ports and management sub-functions:

   kubectl run test --rm -it --image=ubuntu --overrides='{"spec":{"containers":[{"name":"test","image":"ubuntu","securityContext":{"privileged":true}}]}}'

2. Inside the pod, list RDMA devices:

   ls /sys/class/infiniband/
   # Shows: ibp0 ibp1 ... ibp7 mlx5_0 mlx5_1 ... mlx5_8

3. Any UCX consumer (e.g. NIXL createBackend("UCX")) fails:

   uct_iface_open(rc_verbs/mlx5_5:1) failed: Address not valid
   Failed to create UCX worker: Address not valid
   

• UCX version: bundled with nixl-cu13 1.3.1 (libucp/libucs from nixl_cu13.libs)
• UCX_NET_DEVICES not set (default: all)

Setup and versions

• OS: Ubuntu 22.04.5 LTS, kernel 6.8.12-680-6063-coreweave-amd64, x86_64

• RDMA devices:

  Device	PCI	Link	GID[0]	Notes
  ibp0–ibp7	8 separate PCI slots	Ethernet	valid	IB data ports, work fine
  mlx5_0–mlx5_8	0000:9d:00.0–01.0 (1 slot, 9 sub-functions)	Ethernet	all zeros	Management PF + SFs, fail rc_verbs

• GPU: NVIDIA H200, driver 580.126.09, CUDA 13.0

• UCX consumer: NIXL 1.3.1 → UCX (bundled)

Workaround

export UCX_NET_DEVICES=ibp0,ibp1,ibp2,ibp3,ibp4,ibp5,ibp6,ibp7

Works but requires per-node knowledge of which devices are valid. No wildcard/glob support.

Additional information

• Related: Attempts communication over unroutable IP interface #9383 (UCX discovers docker0 in containers, same class of bug for TCP)
• Related: UCT/TCP: Filtered out bridge devices. #9475 (PR adding graceful skipping for TCP interfaces — same pattern should extend to verbs/RDMA)
• All mlx5_* devices report state=ACTIVE, link=Ethernet, type=CA but have all-zero GID tables — a strong pre-filter signal