create baselines #291
Open
create baselines #291
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
Adds a new “milestone” feature that lets tasks define baselines, tracks their runs in the database, and exposes CLI/API/Discord commands for submitting and reporting milestone runs.
- Introduce
milestonesfield on tasks and include it in generated configs - Extend the DB schema and data-access code for milestones and milestone runs
- Update submission logic, Discord cogs, API handlers, and example tasks/workflows to support the new mode
Reviewed Changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| utils.py | Include milestones in task‐config payload |
| task.py | Add milestones field to LeaderboardTask |
| submission.py | Extend prepare_submission signature to take a mode |
| run_eval.py | Treat "milestone" as a valid mode in eval runner |
| report.py | Add short‐report entries for milestone results |
| migrations/…add-milestone-table.py | Create milestones and milestone_runs tables |
| leaderboard_db.py | CRUD methods for milestones and runs + cleanup in delete |
| consts.py | New SubmissionMode.MILESTONE, system‐user helpers |
| cogs/submit_cog.py | Handle milestone submissions end-to-end in Discord cog |
| cogs/leaderboard_cog.py | Import and use lookup, added system user for milestones |
| cogs/admin_cog.py | Commands for listing and submitting milestones, auto-submit |
| api/utils.py | Pass mode into prepare_submission |
| api/main.py | Enrich 404 message when running a missing leaderboard |
| examples/matmul_py/* | Add two reference implementations and milestone definitions |
| examples/eval.py | Treat "milestone" identically to "leaderboard" |
| .github/workflows/nvidia_workflow.yml | Install jq twice (duplicate) |
Comments suppressed due to low confidence (5)
src/discord-cluster-manager/cogs/submit_cog.py:180
- The variable
nameis not defined in this scope. Did you mean to usefilename(the submission filename) instead?
milestone = next((m for m in milestones if m["filename"] == name),
src/discord-cluster-manager/cogs/submit_cog.py:176
- You're passing the tuple returned by
get_system_user_name(None)into theuser_namecolumn. Unpack it so you pass the ID and the name string separately.
(str(SYSTEM_USER_ID), get_system_user_name(None)),
.github/workflows/nvidia_workflow.yml:33
- [nitpick] You install
jqtwice (lines 33 and 36) using bothaptandapt-get. Consolidate to a single install step.
apt update && apt install -y jq
src/discord-cluster-manager/leaderboard_db.py:243
- New methods for milestone CRUD (e.g.
create_milestone,get_leaderboard_milestones,record_milestone_run,get_milestone_runs) should have accompanying unit tests to ensure correct behavior.
def create_milestone(
examples/matmul_py/reference.py:23
- The signature of
check_implementationoriginally returned a single string or empty string; changing it to a tuple may break callers expecting a string. Make sure the consumer is updated to handle(bool, str).
return False, "mismatch found! custom implementation doesn't match reference.: " + reasons[0]
ngc92
reviewed
Jun 15, 2025
| @@ -6,6 +6,20 @@ files: | |||
| - {"name": "utils.py", "source": "../utils.py"} | |||
| - {"name": "reference.py", "source": "reference.py"} | |||
| - {"name": "eval.py", "source": "../eval.py"} | |||
| - {"name": "pytorch_ref.py", "source": "pytorch_ref.py"} | |||
There was a problem hiding this comment.
Milestones shouldn't be added here. These files get sent to the runner for every submission.
There was a problem hiding this comment.
ohh is there another place to config problems or not really?
ngc92
reviewed
Jun 15, 2025
| @@ -366,7 +381,149 @@ async def create_leaderboard_in_db( | |||
| ephemeral=True, | |||
| ) | |||
| return False | |||
| return True | |||
|
|
|||
| # Check if the task has milestones and automatically submit them | |||
There was a problem hiding this comment.
This should be a separate function. create_leaderboard_in_db should do what the name says, having it spawn stuff on runners would be rather surprising.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds milestones to the leaderboard. These are defined in the task.yml of a problem. An example of this can be found in
examples/matmul/task.yml.The idea of these milestones are
This PR primarily makes it such that we trigger runs on leaderboard creation to add milestones to the leaderboard when it is created. It also adds some admin cogs to rerun and view milestones manually (for testing purposes). Below are some screenshots showing things work. (You can also create a leaderboard and test things yourself)
copilot summary
This pull request introduces major updates to the milestone submission system and leaderboard functionality, along with additional enhancements to task files and error handling. The changes aim to streamline milestone management, improve code modularity, and enhance user experience in Discord-based workflows.
Milestone Management and Submission Enhancements:
src/discord-cluster-manager/cogs/admin_cog.py, src/discord-cluster-manager/cogs/admin_cog.pyR384-R527)submit-milestones,list-milestones, andmilestone-resultsto manage and view milestone submissions directly from Discord. (src/discord-cluster-manager/cogs/admin_cog.py, src/discord-cluster-manager/cogs/admin_cog.pyR1185-R1368)Task File Updates:
pytorch_ref.pyandtorch_mm_ref.pyto provide baseline performance implementations for matrix multiplication tasks. (examples/matmul_py/pytorch_ref.py, [1];examples/matmul_py/torch_mm_ref.py, [2]task.ymlto include milestones: Defined milestones for the new reference implementations, enabling automated performance benchmarking. (examples/matmul_py/task.yml, examples/matmul_py/task.ymlR9-R22)Leaderboard Functionality Improvements:
src/discord-cluster-manager/cogs/leaderboard_cog.py, src/discord-cluster-manager/cogs/leaderboard_cog.pyL66-R101)src/discord-cluster-manager/api/main.py, src/discord-cluster-manager/api/main.pyL359-R363)General Improvements:
jqinstallation commands in.github/workflows/nvidia_workflow.yml. (.github/workflows/nvidia_workflow.yml, .github/workflows/nvidia_workflow.ymlR31-R34)src/discord-cluster-manager/cogs/admin_cog.py, [1];src/discord-cluster-manager/cogs/leaderboard_cog.py, [2]These updates collectively enhance the usability and functionality of the milestone and leaderboard systems, making it easier for users to benchmark and manage submissions efficiently.