Skip to content

UCP/GTEST: Add test for multi-rail load-balance - #11842

Open
ZhenlongMa wants to merge 1 commit into
openucx:masterfrom
ZhenlongMa:lb_unit_test
Open

UCP/GTEST: Add test for multi-rail load-balance#11842
ZhenlongMa wants to merge 1 commit into
openucx:masterfrom
ZhenlongMa:lb_unit_test

Conversation

@ZhenlongMa

@ZhenlongMa ZhenlongMa commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What?

Add a stats-enabled unit test for multi-rail load-balance after a device speed-change event. The test verifies that PUT and GET operations can distribute and reallocate transmission data across two rails based on their actual rates.

Why?

The existing unit test for speed-change only verify the configuration without checking the actual data on each rail. As a result, we need additional tests to guarantee that UCX adopt the latest and correct multi-rail speed distribution.

How?

The test records the byte counters for every lane before and after the data transmission and compares them with the expected distribution.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_proto_mock.cc Outdated
size_t expected_bytes,
const std::map<std::string, double> &expected_rates)
{
constexpr double byte_tolerance = 1.0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls align the = on a common column here; the CodeStyle rule is that the longest LHS (const ucp_lane_index_t num_lanes) gets one space before = and the rest align to it, but byte_tolerance and the others don't line up.

Signed-off-by: Zhenlong Ma <zhenlongm@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

size_t expected_bytes,
const std::map<std::string, double> &expected_rates)
{
constexpr double byte_tolerance = 1.0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the per-lane split is computed with round-up fixed-point weights (ucp_proto_multi_scaled_length() uses div_round_up), so the actual bytes on each device are not the exact expected_bytes * rate / total_rate value used here. is a 1-byte tolerance robust across the possible weight rounding, or could it be flaky? if it depends on the specific 52K/38K message sizes, maybe add a short comment noting why these sizes were chosen.

@ZhenlongMa
ZhenlongMa marked this pull request as ready for review August 27, 2026 02:35
@ZhenlongMa
ZhenlongMa marked this pull request as draft August 27, 2026 02:35
@ZhenlongMa
ZhenlongMa marked this pull request as ready for review August 27, 2026 02:35
@roiedanino
roiedanino self-requested a review August 27, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants