Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testgrid-analysis: create issues based on test set flakiness instead of individual tests. #17924

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

siyuanfoundation
Copy link
Contributor

@siyuanfoundation siyuanfoundation commented May 1, 2024

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

#17773 (comment)

Tested locally, example issue: siyuanfoundation#64

If a test set is flaky, e.g. pull-request unit test fails > 10% of the time, it would create a parent issue for the test set, and create sub-issues for the top 5 fail tests.
If the test keep failing, it would append new comment every time the script is run (which is daily), without creating new issues.

@k8s-ci-robot
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ahrtr
Copy link
Member

ahrtr commented May 3, 2024

cc @ArkaSaha30 @jmhbnz @ivanvc

@ivanvc
Copy link
Member

ivanvc commented May 3, 2024

This report looks useful. However, I'm not sure if having it as a GitHub issue makes sense. I feel like we'll just need to close the report issues, which may add to the work of managing issues.

That being said, I agree that there's great value in having the reports more visible rather than nested in a GitHub action run. My suggestion would be to explore Job Summaries. Also, something that would be useful if we decide to use Job Summaries, would be to provide the link to the GitHub action run in the generated issue.

cc. @serathius, as you suggested more visibility of the reports.

@jmhbnz
Copy link
Member

jmhbnz commented May 3, 2024

/retest

@jmhbnz
Copy link
Member

jmhbnz commented May 3, 2024

How spammy do we expect this new issue opening to be? We already have quite a few open flake issues https://github.com/etcd-io/etcd/issues?q=is%3Aopen+is%3Aissue+label%3Atype%2Fflake

We might end up having to continually close issues as duplicates until some existing known flakes are resolved which could be annoying.

Some description of what impact this pr is expected to have would help make a decision on if it will be helpful or harmful.

@siyuanfoundation
Copy link
Contributor Author

I will re-evaluate if this PR is necessary. Mark it as draft for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

None yet

6 participants