Repository navigation
Fix resume mixing evaluator versions in PAIBench motion smoothness (evaluator identity fingerprint) - #375
Fix resume mixing evaluator versions in PAIBench motion smoothness (evaluator identity fingerprint)#375JIN-strong wants to merge 5 commits into
Conversation
Part of NVIDIA#361. Adds pure helpers to fingerprint the effective scorer source (git HEAD + worktree state), checked-in patches, and the environment freeze, with soft degradation when git or freeze are absent. Signed-off-by: JIN-strong <1228766517@qq.com>
Resume now hard-fails when the effective scorer (git source, applied patches, environment freeze) differs from the manifest, so merged scores can no longer mix evaluator versions. Fixes NVIDIA#361. Signed-off-by: JIN-strong <1228766517@qq.com>
Verifies evaluator identity reports unavailable/missing and resume still works under the degraded baseline, and skips git-dependent tests where git is absent. Signed-off-by: JIN-strong <1228766517@qq.com>
Signed-off-by: JIN-strong <1228766517@qq.com>
- legacy RUNNER_VERSION=2 manifests now raise a dedicated message ('legacy manifest without evaluator identity') instead of a misleading 'restore the previous evaluator' error, since no identity was recorded and the evaluator itself may be unchanged; the evaluator-identity mismatch path is preserved for when the key exists but differs.
- hash_patches now folds filename and length into the digest so different patch sets cannot collide (e.g. {a:AB,b:C} vs {a:A,b:BC}).
- add a legacy-manifest rejection test and a collision-regression test.
- README: add git status diagnostic hint for unexpected resume failures.
Signed-off-by: JIN-strong <1228766517@qq.com>
|
Thanks for this, @JIN-strong. The Cosmos cookbooks and evaluation suites have moved to NVIDIA/cosmos-framework under The evaluator-identity fingerprint isn't in Cosmos Framework yet. Could you re-submit it against The relative paths are unchanged, just under |
What
The PAIBench-G motion-smoothness runner (
run_motion_smoothness_sharded.py) recorded onlyevaluator_root(a path) in its resume manifest. After the scorer checkout, its compatibility patches, or the scoring environment change, resuming silently mixed scores computed by different evaluator versions into one aggregated result.This PR fingerprints the effective evaluator into the resume manifest and hard-fails resume when it changes:
build_evaluator_identity()computes an identity from the scorer git HEAD plus agit status --porcelainhash (captures in-place edits and untracked files), content hashes of the checked-in compatibility patches, and thescorer_environment.freeze.txthash.evaluator_identitykey andRUNNER_VERSIONbumps 2 → 3.prepare_manifest()hard-fails with an actionable message when the previous identity differs. Legacy (pre-evaluator_identity) manifests get a dedicated message pointing to a new output directory.status: "unavailable"with a warning, and a missing freeze or patches directory recordsstatus: "missing", so resume protection degrades to the previous path+mtime baseline instead of crashing.Why
Resuming a sharded run after a scorer upgrade could merge scores from the old and new evaluator versions without any signal, producing results that no single evaluator version had actually computed.
Fixes #361.
Validation
python -m pytest test_run_motion_smoothness_sharded.py -qinevaluation/cosmos3/generator/paibench_g→ 14 passed, covering fingerprint determinism, rejection of changed patches or a changed scorer source, legacy-manifest rejection, and the degradation paths. Git-dependent tests skip cleanly wheregitis absent.python -m py_compile run_motion_smoothness_sharded.py→ clean.grep -n "evaluator-root" run_with_cosmos_framework.ipynb→ only the existing--evaluator-root "$PAIBENCH_SCORER_DIR"invocation; no new CLI arguments are required (patches_dirdefaults from__file__).