-
Notifications
You must be signed in to change notification settings - Fork 539
Benchmark: Fix uploading results for jobs finishes with partial success #7989
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I'm picking up this one because it's kind of related to the issue from pytorch/test-infra#6277 where failed benchmark runs weren't uploaded to the database |
@huydhn Is this one closed unintentionally? pytorch/test-infra#6302 seems only address the first part of the issue. |
lol, GitHub is trying hard to be smart here. This is not done yet. |
I think we can close this issue now as the issue has been fixed by pytorch/test-infra#6302 (I thought that I needed to make an additional change on ExecuTorch side, but I was wrong). I did a test in https://github.com/pytorch/executorch/actions/runs/13575020985 to fail some benchmark jobs, but any available benchmark results would still be uploaded (dashboard) I'm waiting for an actual example in trunk to confirm, but perf signals in trunk are mostly green and no such case has showed up yet :) I think we can close this now, and reopen it if needed |
🐛 Describe the bug
This is an orthogonal issue to #7920, where inference runs successfully on high-end iPhone 15 models but fails on the basic iPhone 15. Since all results are reported in the
benchmark-on-device job
, a job that finishes with partial success is marked as a failed job. As a result, none of the results are uploaded or reported to the dashboard. For example, the 'ic4 mps' entry is missing from the JSON file to be uploaded. See details in theupload-benchmark-results
step: https://github.com/pytorch/executorch/actions/runs/12920386457/job/36040740504.Versions
trunk
cc: @kimishpatel @digantdesai
The text was updated successfully, but these errors were encountered: