UCT/IB/MLX5/RC: Avoid caching AH for DevX RoCE QP connect

[tvegas1]

What?

RC's DevX (mlx5) RoCE QP connect no longer uses the iface-wide cached address handle (AH). It creates a temporary, uncached AH just to extract the AV bytes needed for the QP context, then destroys it right away.

Why?

The cached AH is keyed by resolved LID/GID and has no reliable invalidation path, so a peer's stale, no-longer-valid L2 address could keep being reused. The AH is only needed transiently here, to read its AV bytes into the QP context.

How?

uct_rc_mlx5_iface_common_devx_connect_qp() now creates the AH on RoCE and destroys it immediately after extracting the AV bytes, instead of going through AH cache. The fix covers every caller: RC (Vebs unaffected), GDAKI, GGA, and the internal tag-matching command QP.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Minor consistency note: the warn-on-failure pattern for ibv_destroy_ah is new here — elsewhere it's called without a check (e.g. uct_ib_device_test_roce_gid_index and cache cleanup). Warning on failure is reasonable and matches uct_ib_destroy_cq, so not a concern.

Minor (non-blocking): the ret value passed into ucs_warn("... returned %d: %m", ret) is redundant with %m (both convey the errno), but this mirrors existing style (uct_ib_device_query's ibv_query_port warning), so it's fine as-is.

Test coverage note: this is a behavioral change on the RoCE devx QP-connect path (AH is now transient rather than cached). It relies on existing RoCE + devx connection-establishment coverage rather than a new focused test. Worth confirming the CI matrix includes a RoCE + devx job so the transient-AH create/destroy path is actually run.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer BlueField on worker 1) · commit 4398d83f

TL;DR: The gtest suite passed completely (8609/8609); the job failed only on a LeakSanitizer report of a 131-byte glibc-internal dlerror() message buffer allocated from ucs_sys_get_lib_info() at src/ucs/sys/lib.c:18. Fix by deleting the pointless (void)dlerror(); call there (and/or adding leak:dlerror to contrib/lsan.supp) — this is unrelated to PR #11828.

Full analysis

Summary: AddressSanitizer BlueField job failed at test teardown with ERROR: LeakSanitizer: detected memory leaks — SUMMARY: AddressSanitizer: 131 byte(s) leaked in 1 allocation(s), causing make: *** [Makefile:4713: test] Error 1, even though all 8609 gtest cases reported [ PASSED ].

Root cause: The leaked block is allocated inside glibc itself, not UCX:

#1 __vasprintf_internal libio/vasprintf.c:71
#2 ___asprintf stdio-common/asprintf.c:31
#3 __dlerror dlfcn/dlerror.c:74
#4 ucs_sys_get_lib_info  src/ucs/sys/lib.c:18
#5 ucs_sys_get_lib_path  src/ucs/sys/lib.c:32
#6 ucs_profile_write     src/ucs/profile/profile.c:322

ucs_sys_get_lib_info() starts with an unconditional (void)dlerror(); (line 18). glibc's __dlerror() only heap-allocates when a dynamic-linking error is pending — it asprintf()s the message string, and that string is freed only on the next dlerror() call or at thread exit. UCX's module loader (ucs_module_try_load/ucs_module_dlsym_shallow in src/ucs/sys/module.c:261-267, 150-162) routinely leaves failed dlopen/dlsym errors pending on BlueField (rocm/cuda/optional plugins that don't exist there — visible in the log as rocm_copy variants and the suppressed dlsym entry). The dlerror() at lib.c:18 is the call that materializes that pending message into a 131-byte heap string, and since nothing ever calls dlerror() again before exit, LSAN reports it as leaked at ucs_profile_cleanup (test/gtest/ucs/test_profile.cc:48).

Critically, the dlerror() at line 18 is dead code: the error path at lines 20-22 returns UCS_ERR_NO_MEMORY and never consults dlerror(), so clearing the error state buys nothing. contrib/lsan.supp suppresses dlopen, dlsym, bfd_map_over_sections, ibv_alloc_pd, etc., but has no entry for dlerror — hence the gap. This is environment-dependent (it only fires when a dl error happens to be pending at that moment), which makes it a flaky failure rather than a regression.

Implicated commit: Not PR #11828. The leaking line predates it (src/ucs/sys/lib.c last touched by bca0bf5b, Alexey Rivkin, 2022-07-27 — copyright-only; logic from 125542b1, Leonid Genkin). The exposure was most likely widened by 49a0d4c1 (Roie Danino, "UCS/SYS: Added support for dynamically loaded external modules/plugins (#11206)"), which added the external plugin-path dlopen loop and last edited contrib/lsan.supp. PR #11828 (UCT/IB/MLX5/RC: Avoid caching AH for DevX RoCE QP connect) touches RC/DevX address-handle code and cannot produce this stack.

File: src/ucs/sys/lib.c:18 (leak site); contrib/lsan.supp:1-7 (missing suppression); triggered via src/ucs/profile/profile.c:322

Suggested fix:

1. Preferred — remove the useless call in src/ucs/sys/lib.c, which eliminates the allocation entirely:

   ucs_status_t ucs_sys_get_lib_info(Dl_info *dl_info)
   {
       /* NOTE: do not call dlerror() here - on glibc it asprintf()s a pending
        * error message that is only freed on the next dlerror()/thread exit,
        * which LeakSanitizer reports as a leak. The error string is unused. */
       if (dladdr(ucs_sys_get_lib_info, dl_info) == 0) {
           return UCS_ERR_NO_MEMORY;
       }
       return UCS_OK;
   }

2. Belt-and-braces — add a suppression so any other glibc dlerror buffer can't fail the ASAN job:

   leak:dlerror
   

   appended to contrib/lsan.supp.
3. Consider auditing the other bare (void)dlerror(); calls in src/ucs/sys/module.c (lines 70, 150, 157) — those are legitimate (their errors are consumed by the adjacent dlerror() in the log message), but the final message string of the last failing dlopen in ucs_module_try_load has the same latent leak shape.
4. Re-run the BlueField ASAN job on PR UCT/IB/MLX5/RC: Avoid caching AH for DevX RoCE QP connect #11828 once the suppression/fix lands; nothing in this log indicates a defect in the PR.

Related: PR #11828 (the PR under test — unaffected); PR #11206 (added external plugin loading + last lsan.supp change). No existing issue tracks the dlerror LSAN leak — worth opening one.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Residual coverage gap (already flagged in prior review): this is a behavioral change on the RoCE + devx QP-connect path relying on existing connection-establishment coverage; worth confirming a RoCE + devx CI job actually exercises the transient create/destroy path.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests roce on worker 2) · commit 6f4fb0c8

TL;DR: The roce on worker 2 gtest job died because a UMR memory-window bind on the MD's self-connected UMR QP never completed (30 s ibv_poll_cq timeout in ib_mlx5dv_md.c:1443), making ucp_rkey_pack() fail with I/O error in ib/test_ucp_proto_emulation_enable.no_zcopy_proto_fails_get_small/0; this is in the MD-level UMR RoCE loopback path, not in the AH code PR #11828 touches, and the extra 16 min of runtime is a test-teardown spin loop with no deadline.

Full analysis

Summary: ib/test_ucp_proto_emulation_enable.no_zcopy_proto_fails_get_small/0 failed with Error: Input/output error after mlx5_0: ibv_poll_cq(UMR CQ, registration, UMR mkey ... index 0x3cb8) timed out, then hung ~15 min in EP-close teardown until "Connection timed out - abort testing" aborted gtest (SIGABRT, make: *** [Makefile:4713: test] Aborted).

Root cause: Evidence chain from the log:

• 09:03:19 test starts; 09:03:49 (exactly 30 s later) ib_mlx5dv_md.c:1443 reports the UMR CQ poll timeout — that is the 30 s bound in uct_ib_mlx5_devx_umr_post_sync(). The posted work request is the IBV_WR_BIND_MW from uct_ib_mlx5_devx_umr_mkey_bind() (ib_mlx5dv_md.c:1542), sent on md->umr.qp, an RC QP self-connected over the local port. The completion never arrived.
• The follow-up ibv_post_send(UMR QP, invalidation, ...) returned 12: Resource temporarily unavailable is secondary — the send queue is still occupied by the stuck WQE.
• Registration failure propagates to ucp_rkey_pack() → ASSERT_UCS_OK at test/gtest/ucp/ucp_test.cc:1281.
• 09:06:50 → 09:18:50: request 0x... did not complete on time, then 12 min of silence with zero output while ucp_test_base::entity::close_all_eps() spins in while (!is_request_completed(req)) test.progress(); (ucp_test.cc:917-921) — the inner loop has no deadline check, so a request that can never complete burns the whole global timeout. That is the hang, and it is teardown fallout, not the primary fault.

The UMR QP self-connect is RoCE-fragile: uct_ib_mlx5_devx_umr_modify_qp() hardcodes is_global = 1 with UCT_IB_DEVICE_DEFAULT_GID_INDEX (GID 0 = RoCEv1) and does not honour the configured GID index / RoCE version / DSCP-sport selection used elsewhere for RoCE ifaces, so on a RoCEv2-configured port the loopback packet can be silently dropped instead of completing. This is consistent with the failure appearing only in the RoCE job.

PR #11828 (UCT/IB/MLX5/RC: Avoid caching AH for DevX RoCE QP connect, src/uct/ib/mlx5/rc/rc_mlx5_devx.c:415-449) is very unlikely to be the trigger: it only affects uct_rc_mlx5_iface_common_devx_connect_qp(), which copies the AV into the QPC before ibv_destroy_ah(); the failure occurred in MD-level memory registration/rkey packing, before any RC data traffic, and the UMR QP is a plain verbs QP that does not use that AH path at all.

Implicated commit: unknown for the UMR failure (UMR export path predates this PR; most recent touches: a956862 "UCT/IB: Support relaxed-only memory keys" by Roie Danino, a47066b by Raul Akhmetshin). PR head 6f4fb0c (Thomas Vegas) is not implicated by the log evidence.

File: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:1441-1448 (30 s UMR poll timeout) and :1218-1227 (UMR QP RoCE AH attrs, hardcoded GID index 0); test/gtest/ucp/ucp_test.cc:917-921 (deadline-less teardown loop)

Suggested fix:

1. Re-run the job to confirm the UMR timeout is not reproducible for this PR; if it reproduces only on swx-rain03, check that node's port/GID configuration and firmware (UMR loopback traffic on RoCEv1 GID 0).
2. Harden the UMR QP RoCE self-connect: select the GID index/RoCE version the same way the IB iface does (md->super.config.gid_index / uct_ib_device_select_gid + RoCEv2 UDP sport) instead of hardcoding UCT_IB_DEVICE_DEFAULT_GID_INDEX with is_global = 1 in uct_ib_mlx5_devx_umr_modify_qp(), and disable the UMR export path (graceful fallback) if the initial self-loopback probe does not complete, rather than timing out on every registration.
3. Add a ucs::get_deadline() check to the inner while (!is_request_completed(req)) loop in close_all_eps() so a failed request aborts the test in seconds instead of consuming 15 minutes of CI wall time.

Related: PR #11828 (the branch under test); #11433 (previous work on this test); UMR/mkey changes #11649