Skip to content

Fix NVML mismatch recovery detection - #1522

Closed
bigboateng wants to merge 1 commit into
gazebo-tooling:masterfrom
bigboateng:fix/nvml-recovery-detection
Closed

Fix NVML mismatch recovery detection#1522
bigboateng wants to merge 1 commit into
gazebo-tooling:masterfrom
bigboateng:fix/nvml-recovery-detection

Conversation

@bigboateng

Copy link
Copy Markdown

Summary

  • Match the stable NVML driver/library version mismatch diagnostic
    independently of runtime prefix and capitalization.
  • Keep the Naginator retry expression and post-build recovery guard aligned.

Root cause

The recovery logic expected the older, case-sensitive message:

nvml error: driver/library version mismatch

The NVIDIA container runtime now reports:

failed to initialize NVML: Driver/library version mismatch

As a result, affected GPU jobs failed before container startup and the
post-build script incorrectly skipped automatic recovery:

Validation

  • Generated all Jenkins DSL XML with OpenJDK 17.
  • Ran jenkins-scripts/dsl/dsl_checks.bash.
  • Ran jenkins-scripts/docker/sanity_checks.bash.
  • Confirmed the generated _test_job_from_dsl.xml uses the updated expression
    in both Naginator and the post-build Groovy guard.
  • Confirmed with Java regular expressions that the old expression misses the
    reported runtime message while the updated expression matches both the old
    and reported forms.

The remaining integration check is to run the safe _dsl_test Jenkins fixture
with the reported message and verify retry, recovery relabeling, and reboot
behavior.

Signed-off-by: bigboateng <bigboateng2011@gmail.com>
@j-rivero

Copy link
Copy Markdown
Contributor

better approach in #1525.

@j-rivero j-rivero closed this Jul 30, 2026
@bigboateng
bigboateng deleted the fix/nvml-recovery-detection branch July 31, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants