Use bazel CI to repeat tests and identify flakes

[scpeters]

🎉 New feature

Similar to gazebosim/gz-math#820

Summary

New workflows focused on detecting and characterizing flaky tests

This adds workflows that can run tests repeatedly in bazel to gain information about flaky tests. Since bazel targets can be annotated as flaky (see #866 for an example), distinct workflows are created to execute flaky and non-flaky tests separately.

Any failures of "non-flaky" tests indicate that the offending test should be annotated as flaky. The results from repeated executions of flaky tests provides flakiness statistics
and may aid in diagnosing the root cause of flakiness.

The workflows are implemented with a parameterized Bazel repeat tests workflow that is invoked via workflow_call by the Bazel repeat flaky tests and Bazel repeat non-flaky tests workflows. For testing, the initial version (6624e08) of the flaky and non-flaky workflows are triggered on push to this branch in order to confirm that the workflows run properly, and are adjusted in 7d72098 to run on a schedule on Mondays and Fridays. I believe GitHub purges unused caches every 7 days, so this frequency prevents the bazel cache from being purged.

Each workflow will be manually invokable via workflow_dispatch once this PR is merged, with the runs_per_test parameter exposed by each workflow, with a default value of 10.

Other fixes

In initial testing of this branch, I noticed that Clock_TEST and Node_TEST are flaky when executed in this workflow, so I annotated them as flaky (4da1dd2) and then additionally added the exclusive tag to Clock_TEST (a5ad8d2) to prevent that test to be executed in parallel since it seems particularly sensitive to system load.

Backport Policy

• This is safe to backport to the following versions:
  • Jetty
  • Ionic
  • Harmonic
  • Fortress
• This should not be backported
• I am not sure
• Other (fill in yourself)

We can backport this if we want, but the scheduled workflows will only run from main.

Test it

Review the CI results from 1fdf01d

• Repeat flaky tests result completed in 6m40s and detected several failures.
• Repeat non-flaky tests result was successful in 4m23s, executing all bazel tests 10 times

Checklist

• Signed all commits for DCO
• Added a screen capture or video to the PR description that demonstrates the feature
• Added tests
• Added example and/or tutorial
• Updated documentation (as needed)
• Updated migration guide (as needed)
• Consider updating Python bindings (if the library has them)
• codecheck passed (See contributing)
• All tests passed (See test coverage)
• Updated Bazel files (if adding new files). Created an issue otherwise.
• While waiting for a review on your PR, please help review another open pull request to support the maintainers
• Was GenAI used to generate this PR? If so, make sure to add "Assisted-by" to your commits. (See this policy for more info.)

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by and Generated-by messages.

Backports: If this is a backport, please use Rebase and Merge instead.

[caguero]

Thanks, I left a few comments.

[caguero]

bazel.yml uses -c opt to avoid perf related flakiness. This workflow uses fastbuild, so it tests different binaries and fills the disk cache with fastbuild artifacts. Should we add -c opt here for consistency?

[caguero]

This step runs with bash -e and no pipefail. If bazel query itself fails, grep gets no input and the job exits 0 with no tests run. It also runs the query twice. Suggestion:

Suggested change

	# Exit with success code 0 if no targets match the query
	# `grep -q '^' fails if the query output is empty
	bazel query "attr(flaky, ${{ inputs.flaky }}, tests(//...))" \
	| grep -q '^' || exit 0
	bazel test --runs_per_test=${{ inputs.runs_per_test }} \
	--test_output=errors \
	$(bazel query "attr(flaky, ${{ inputs.flaky }}, tests(//...))")
	# Exit with success code 0 if no targets match the query
	targets=$(bazel query "attr(flaky, ${{ inputs.flaky }}, tests(//...))")
	[ -n "$targets" ] || { echo "No matching test targets"; exit 0; }
	bazel test --runs_per_test=${{ inputs.runs_per_test }} \
	--test_output=errors \
	$targets

[caguero]

Nit: the defaults are numbers but the type is string. Should we use type: number?

[caguero]

Nit: this comment was copied from test/BUILD.bazel. I think for these targets it should be //:all or //....

[shameekganguly]

nit: maybe expand on why this test should run exclusively?