Skip to content

Summarize XPASS-ing tests in CI #714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Mar 7, 2025
Merged

Conversation

sgillies
Copy link
Collaborator

@sgillies sgillies commented Feb 27, 2025

We're xfailing some tests that have a history of flakiness or timing out, but it's difficult to see when they xpass in the GitHub UI. You have to click through each job, find the step and scroll through it. And we've got a lot of jobs.

I've tried to add a summary job. It's not very fancy (I'm having trouble reproducing some unexpected grep behavior on my laptop), but it does show every xfailing test with information that helps guide us back to the particular job.

For example, this

XPASSes:
test-outputs/partial-test-output-tiledb-3.9-pinned-ubuntu-24.04-2/test.log:tests/taskgraphs/test_server_side.py::ConnectToExistingTest::test_running_graph �[33mXPASS�[0m
test-outputs/partial-test-output-tiledb-3.9-pinned-ubuntu-24.04-2/test.log:=================================== XPASSES ====================================
test-outputs/partial-test-output-tiledb-3.9-pinned-ubuntu-24.04-2/test.log:�[33mXPASS�[0m tests/taskgraphs/test_server_side.py::�[1mConnectToExistingTest::test_running_graph�[0m - Server side error, graph not computed in expected time.
test-outputs/partial-test-output-tiledb-3.9-pinned-ubuntu-24.04-4/test.log:tests/test_dag.py::DAGClassTest::test_resource_class �[33mXPASS�[0m (Requires...)
test-outputs/partial-test-output-tiledb-3.9-pinned-ubuntu-24.04-4/test.log:=================================== XPASSES ====================================
test-outputs/partial-test-output-tiledb-3.9-pinned-ubuntu-24.04-4/test.log:�[33mXPASS�[0m tests/test_dag.py::�[1mDAGClassTest::test_resource_class�[0m - Requires resolution of upstream issue.

tells us that two tests unexpectedly passed. Both with Python 3.9 (the Python 3.12 tests don't exercise UDFs or task graphs), pinned dependencies. One test is part 2 of the batch of splits, one in part 4.

In the UI, it looks like this:

summary

The history of this PR is a mess. The net changes are small.

@sgillies sgillies changed the title Sg/sc 63777/summarize test jobs Summarize XPASS-ing tests in CI Mar 6, 2025
@sgillies sgillies marked this pull request as ready for review March 6, 2025 21:34
@sgillies sgillies force-pushed the sg/sc-63777/summarize-test-jobs branch from 8bf5114 to 87f708e Compare March 6, 2025 21:46
path: .test_durations
key: ${{ env.PYTEST_TEST_DURATIONS_CACHE_KEY }}-${{ hashFiles('.test_durations') }}
echo "XPASSes:" >> $GITHUB_STEP_SUMMARY
grep --recursive --with-filename "XPASS" test-outputs >> $GITHUB_STEP_SUMMARY
Copy link
Collaborator Author

@sgillies sgillies Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been unable to get "^XPASS" to work as a pattern, neither with nor without --extended-regexp. In the GHA runner, it matches nothing.

@ktsitsi ktsitsi self-requested a review March 7, 2025 16:37
@sgillies sgillies assigned sgillies and unassigned ktsitsi and spencerseale Mar 7, 2025
@sgillies sgillies merged commit cc41e0d into main Mar 7, 2025
18 checks passed
@sgillies sgillies deleted the sg/sc-63777/summarize-test-jobs branch March 7, 2025 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants