UCP: Tune Alltoall pairwise posting depth

[jeffnvidia]

What

Replace the binary automatic choice of posting either one or all peer transfers with a coarse posting-depth selector. Explicit ALLTOALL_PAIRWISE_NUM_POSTS values remain authoritative.

if total_bytes <= 66000 or N <= 8: P = N
else if peer_bytes <= 64 KiB:       P = min(N, 32)
else if peer_bytes <= 1 MiB:        P = min(N, 16)
else if peer_bytes <= 4 MiB:        P = N for N <= 16, otherwise 8
else if peer_bytes <= 8 MiB:        P = min(N, 4)
else:                               P = 4 for N <= 32, otherwise 1

The existing 66,000-byte aggregate-message behavior is retained. The exact 64 KiB boundary is a separate per-peer threshold. Per-peer bytes use the same calculation as the progress path.

Why

The current automatic selector uses one outstanding send and receive for every team larger than 32 ranks once the aggregate message exceeds 66,000 bytes, while teams of at most 32 ranks post the full team. The former serializes small and medium peer transfers too aggressively; the latter can over-post medium and large transfers.

NUM_POSTS=P limits incomplete sends and receives independently, so intermediate depths provide useful admission control without changing peer order or the UCP data path.

Evidence

All performance evidence below uses streaming cadence. The rule was developed from independent multi-rank, multi-protocol tuning corpora, then checked in held-out, order-balanced, transfer, and 128-rank validation. Performance comparisons use the explicit depth selected by this rule against the explicit depth selected by the previous automatic rule; focused unit tests independently verify the automatic mapping at every boundary.

Retrospective and transfer corpora

• A stable-protocol 4/8/16/32/64-rank corpus has 55 matched cells: +23.5% geometric mean versus the current selector, no matched regression, and geometric-mean oracle coverage improving from 76.5% to 94.4%.
• A default-protocol corpus has 44 matched cells: +22.0% geometric mean with no matched regression. Eleven cells are excluded because the required candidate did not complete in an older crash-affected sweep.
• Two newer independent 32/64-rank streaming corpora score +34.5% and +36.1% geometric mean, respectively, with no matched regression.
• A 32-rank/PPN4 transfer validation independently tuned posting depth for the default schedule and for topology-aware peer order. The topology-aware schedule improved every 16--128 MiB-per-peer cell by +2.66% to +10.90% with 95% confidence intervals excluding zero. The selected depth differs sharply by schedule at 16--64 MiB, while both select P1 at 128 MiB, confirming that peer ordering and posting depth are separate, interacting controls.

The rule intentionally preserves full posting for teams of eight ranks or fewer: throttling those teams did not provide a robust cross-protocol benefit, and caused 5--25% regressions above 8 MiB per peer.

Held-out and position-balanced validation

Fresh held-out validation used PPN8, streaming cadence, default UCP protocol selection, and nine non-power-of-two sizes from 48 KiB through 12 MiB per peer.

• At 64 ranks, the selected depth improves geometric-mean bandwidth by 104.1% over the old selector across all nine cells, with no regression (individual changes: 0.0% to +429.3%).
• At 32 ranks, an initial six-way sequence was strongly position/bimodality sensitive. Four focused six-repeat scouts rotated every relevant configuration through every sequence position. Replacing those four noisy cells with the position-balanced measurements gives +22.3% geometric mean across nine cells, with no regression. The rotated 48/96/384/1536-KiB-per-peer cells were 0.0%, +9.1%, +28.8%, and +10.9% versus the old selector.
• 211 required timing samples parsed with zero required failure. One optional P4 stress control at 64 ranks and 12 MiB per peer timed out; both the old and proposed selectors use P1 there and completed. Two of 66 position-rotated launches hit an existing UCX lanes_distance assertion before producing a timing; the other 64 launches completed and those infrastructure failures are excluded.

Fresh 128-rank confirmation

A focused PPN4 128-rank, 64-MiB-aggregate / 512-KiB-per-peer validation balanced six repetitions of every candidate across sequence positions.

• With default protocol selection, all 78/78 cases completed. Proposed P16 is the six-run median winner at 1.675 GB/s, +28.35% versus current auto/P1 at 1.305 GB/s, and beats auto in all six repetitions.
• With a protocol-matched legacy configuration, all 78/78 cases completed. P35 is raw-best at 1.720 GB/s and P4 is within 2% at 1.690 GB/s; proposed P16 reaches 1.475 GB/s, but still improves +49.75% versus auto/P1 at 0.985 GB/s and beats auto in all six repetitions.

This focused result removes the previous single-sweep concern around P16. It is positive confirmation for the proposed rule, rather than a broad no-regression claim for that environment.

Impact

This changes only automatic posting-depth selection in the TL/UCP pairwise Alltoall algorithm. It does not change Alltoallv, topology-aware peer ordering, or user-configured posting depths.

Checks

• Added focused tests for every inclusive threshold, the 8/9, 16/17, and 32/33-rank transitions, valid range, and monotonicity.
• Exact-head full UCC/UCX/CUDA build and both focused tests pass.
• make dist passes, and the resulting source tarball contains the new selector header.
• The new header is listed in TL/UCP's Automake source manifest.
• All 16 exact-head GitHub checks pass: normal/ASAN builds and tests, codestyle, docs, standard/CUDA/ROCm clang-tidy, UCC perftest, and OMPI build/IMB.
• Compiled the changed source separately with UCC warning-as-error flags.
• Ran the selector boundary harness locally.
• git diff --check origin/master...HEAD.