Skip to content

Benchmark: Fix uploading results for jobs finishes with partial success #7989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
guangy10 opened this issue Jan 28, 2025 · 4 comments · Fixed by pytorch/test-infra#6302
Closed
Assignees
Labels
enhancement Not as big of a feature, but technically not a bug. Should be easy to fix module: benchmark Issues related to the benchmark infrastructure triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Milestone

Comments

@guangy10
Copy link
Contributor

guangy10 commented Jan 28, 2025

🐛 Describe the bug

This is an orthogonal issue to #7920, where inference runs successfully on high-end iPhone 15 models but fails on the basic iPhone 15. Since all results are reported in the benchmark-on-device job, a job that finishes with partial success is marked as a failed job. As a result, none of the results are uploaded or reported to the dashboard. For example, the 'ic4 mps' entry is missing from the JSON file to be uploaded. See details in the upload-benchmark-results step: https://github.com/pytorch/executorch/actions/runs/12920386457/job/36040740504.

Versions

trunk

cc: @kimishpatel @digantdesai

@guangy10 guangy10 added enhancement Not as big of a feature, but technically not a bug. Should be easy to fix module: benchmark Issues related to the benchmark infrastructure labels Jan 28, 2025
@manuelcandales manuelcandales added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jan 28, 2025
@guangy10 guangy10 moved this to To triage in ExecuTorch DevX Feb 4, 2025
@guangy10 guangy10 moved this from To triage to Ready in ExecuTorch DevX Feb 4, 2025
@huydhn huydhn moved this to Cold Storage in PyTorch OSS Dev Infra Feb 4, 2025
@guangy10 guangy10 added this to the 0.6.0 milestone Feb 10, 2025
@huydhn huydhn moved this to Ready in ExecuTorch Benchmark Feb 11, 2025
@huydhn huydhn moved this from Ready to In Progress in ExecuTorch Benchmark Feb 18, 2025
@huydhn
Copy link
Contributor

huydhn commented Feb 18, 2025

I'm picking up this one because it's kind of related to the issue from pytorch/test-infra#6277 where failed benchmark runs weren't uploaded to the database

@guangy10
Copy link
Contributor Author

@huydhn Is this one closed unintentionally? pytorch/test-infra#6302 seems only address the first part of the issue.

@huydhn
Copy link
Contributor

huydhn commented Feb 19, 2025

lol, GitHub is trying hard to be smart here. This is not done yet.

@huydhn huydhn reopened this Feb 19, 2025
@github-project-automation github-project-automation bot moved this from Done to Backlog in ExecuTorch DevX Feb 19, 2025
@github-project-automation github-project-automation bot moved this from Backlog to Done in ExecuTorch DevX Feb 19, 2025
@guangy10 guangy10 reopened this Feb 19, 2025
@github-project-automation github-project-automation bot moved this from Done to Backlog in ExecuTorch DevX Feb 19, 2025
@guangy10 guangy10 moved this from Done to In Progress in ExecuTorch Benchmark Feb 20, 2025
@huydhn
Copy link
Contributor

huydhn commented Feb 27, 2025

I think we can close this issue now as the issue has been fixed by pytorch/test-infra#6302 (I thought that I needed to make an additional change on ExecuTorch side, but I was wrong). I did a test in https://github.com/pytorch/executorch/actions/runs/13575020985 to fail some benchmark jobs, but any available benchmark results would still be uploaded (dashboard)

I'm waiting for an actual example in trunk to confirm, but perf signals in trunk are mostly green and no such case has showed up yet :)

I think we can close this now, and reopen it if needed

@huydhn huydhn closed this as completed Feb 27, 2025
@github-project-automation github-project-automation bot moved this from Backlog to Done in ExecuTorch DevX Feb 27, 2025
@github-project-automation github-project-automation bot moved this from In Progress to Done in ExecuTorch Benchmark Feb 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Not as big of a feature, but technically not a bug. Should be easy to fix module: benchmark Issues related to the benchmark infrastructure triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
Status: Done
Status: Done
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants