Multiple Test Runs Causing Ambiguous Test Results in Jenkins

[tchaikov]

Issue Summary

When a test fails in our CI pipeline, Jenkins sometimes displays incorrect test results due to multiple test runs having identical identifiers. This makes it difficult to locate and analyze the actual failed test run.

Background

• We modified our CI configuration to run test.py multiple times, stopping on the first failure
• Reference commit: https://github.com/scylladb/scylla-pkg/commit/9610913a89f5c4452d52c5357e3684d04f6947cd

Problem Details

1. Multiple test runs use identical command line arguments for test.py
2. This results in multiple test results sharing the same identifier
3. Jenkins cannot differentiate between successful and failed runs of the same test
4. When accessing a failed test report from the CI job landing page, Jenkins may incorrectly link to a successful run instead

Example Case

• Job: https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/15443/
• Test: "test_backup_to_non_existent_bucket.debug"
• Ran 6 times, with one failure marked as "Regression"
• Actual failure report location: https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/15443/testReport/junit/object_store/test_backup/test_backup_to_non_existent_bucket_debug_6_6/
• Related issue: jenkins cannot tell different tests apart if they are excersing the same test but with different settings #15973

Impact

• Decreased reliability of test result tracking
• Additional time spent locating actual failure reports

[tchaikov]

cc @yaronkaikov @xtrey

[xtrey]

@tchaikov you check allure report that does not have such problem. However, it lacks the link to the directory with scylla logs.
@yaronkaikov I think this is an issue of the executing the tests case 100 in a bunch of 10 runs with 10 repeats as a workaround. As for now, we have an option --max-failures I think we can reed off this batched execution, so the issue will go away.

[yaronkaikov]

@tchaikov you check allure report that does not have such problem. However, it lacks the link to the directory with scylla logs. @yaronkaikov I think this is an issue of the executing the tests case 100 in a bunch of 10 runs with 10 repeats as a workaround. As for now, we have an option --max-failures I think we can reed off this batched execution, so the issue will go away.

We no longer run those tests 100 times but limit them to an hour max. also --max-failures is set for 20

[xtrey]

We no longer run those tests 100 times but limit them to an hour max. also --max-failures is set for 20

--max-failures 20 not very useful in that specific case. We have three tests with repeat 10, so the total is 30 cases. --max-failures 20 is good for big bunches of almost non-repeating tests.
Ok, it's not 100 times but limited by 1 hour. But still, it runs the command with --repeat=10 six times. Jenkins JUnit plugin can't handle this correctly, since there will be six packs of tests with the same name. And a good old issue #15973 is back.

The fix here can be is do not use batches or override the suite name for junit. So every batch can be in its own suite, so there will be no clashes in names.
We can't run tests during specific time with test.py, so removing batching probably will be tricky. Why not to give a shot with overriding the suite names?