UCP/CORE: Allow auxiliary transports for AM fallback

[brminich]

What?

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

The aux fallback interacts with the max_bcopy retry loop. With aux_fallback=1, allow_aux=0, when a primary transport is found but has too-small max_bcopy, it hits continue and re-runs primary-only selection (still allow_aux=0), looping again — the fallback to aux is only triggered by UCS_ERR_UNREACHABLE, never by the max_bcopy path. This means a viable aux transport is never tried when the only primary is rejected for small max_bcopy. Is the aux fallback supposed to also kick in when the only primary AM lane is rejected due to small max_bcopy? Currently that path just retries primary-only and never reaches the allow_aux=1 fallback.

[svc-nvidia-pr-review]

num_usable_tls excludes UCT_DEVICE_TYPE_ACC, but num_primary_tls counts every non-AUX resource including ACC. The combined (num_usable_tls == 0) || (num_primary_tls == 0) check is conservative and still correct, but the asymmetry is easy to misread. A one-line comment on why num_primary_tls intentionally counts ACC would help future readers.

[svc-nvidia-pr-review]

The reworded AUX flag comment ("may be used only when auxiliary transports are explicitly allowed by lane selection") is accurate, but the flag is still also used to gate keepalive and wireup-aux lane selection (criteria.tl_rsc_flags = UCP_TL_RSC_FLAG_AUX). Consider noting both the AM-fallback and wireup/keepalive uses so the comment does not read as AM-only.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 1) · commit a60c0acc

TL;DR: The single gtest failure cuda/test_ucp_cuda_alias.disables_only_cuda_tls/0 is a broken new test: its modify_config("TLS", "^cuda") is silently overwritten by the variant's TLS ("cuda") inside entity::entity(), so CUDA transports (and all:aux-flagged transports) are still present and the EXPECT_FALSE() checks fire. Fix the test to pass ^cuda through the test param (or as a separate UCP_INSTANTIATE_TEST_CASE_TLS variant) instead of modify_config.

Full analysis

Summary: ASAN gpu job failed with exactly 1 failing test out of 10134 — cuda/test_ucp_cuda_alias.disables_only_cuda_tls/0, where GetParam() = cuda — causing make test to exit non-zero (Makefile:4713: test Error 1). No sanitizer report, no hang (log is continuous, longest test 21 s).

Root cause: Test-level bug, not a library bug.

• test_ucp_cuda_alias::disables_only_cuda_tls (test/gtest/ucp/test_ucp_context.cc:230) does modify_config("TLS", "^cuda") and then create_entity(), asserting that no CUDA transport and no UCP_TL_RSC_FLAG_AUX resource exists.
• But ucp_test_base::entity::entity() unconditionally re-applies the variant's transports right before ucp_init: ss << test_param.transports; ucp_test::set_tls(ucp_config, ss.str()); (test/gtest/ucp/ucp_test.cc:665-668, set_tls → ucp_config_modify(config,"TLS",...) at ucp_test.cc:422). Since the instantiation is UCP_INSTANTIATE_TEST_CASE_TLS(test_ucp_cuda_alias, cuda, "cuda"), the effective config is UCX_TLS=cuda, not ^cuda.
• With UCX_TLS=cuda the alias table { "cuda", { "cuda_copy", "cuda_ipc", "gdr_copy", UCP_TL_AUX(UCP_RSC_CONFIG_ALL) } } (src/ucp/core/ucp_context.c:786-787) enables cuda_copy/cuda_ipc/gdr_copy as primary and every other transport with UCP_TL_RSC_FLAG_AUX (ucp_context.c:1263-1267). Hence both EXPECT_FALSE(is_cuda_transport(...)) and EXPECT_FALSE(resource->flags & UCP_TL_RSC_FLAG_AUX) fail. The sibling test_ucp_cuda_alias.resources passes precisely because it expects that (aux) state.
• Secondary latent issue in the same area: ucp_fill_aux_tls() (ucp_context.c:1888-1912) inserts the literal string "all" into aux_tls for the all:aux alias entries instead of expanding to real TL names, so the deny-list branch at ucp_context.c:1272-1299 will never classify real transports as aux-capable for the cuda/cuda_ipc aliases — worth verifying once the test actually applies ^cuda.

Implicated commit: a60c0ac — Mikhail Brinskii, "UCP/CORE: Allow auxiliary transports for AM fallback" (PR #11848 head; touches both src/ucp/core/ucp_context.c and test/gtest/ucp)

File: test/gtest/ucp/test_ucp_context.cc:230-248 (assertions at :242-243); override at test/gtest/ucp/ucp_test.cc:665-668

Suggested fix: Stop relying on modify_config("TLS", ...) in these alias tests, because entity creation resets TLS. Either:

1. Pass the negated list through the test param:

ucp_test_param param = GetParam();
param.transports     = {"^cuda"};
entity *e            = create_entity(false, param);

2. Or add a dedicated instantiation, e.g. UCP_INSTANTIATE_TEST_CASE_TLS(test_ucp_cuda_alias_negate, cuda_neg, "^cuda"), and drop the modify_config call.
   Apply the same change to test_ucp_cuda_ipc_alias.disables_only_cuda_ipc (test_ucp_context.cc:166-189), which has the identical defect and will fail on which