Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bump nvidia driver for CUDA 12.1 support #16557

Open
ddelange opened this issue May 10, 2024 · 4 comments · May be fixed by #16560
Open

Bump nvidia driver for CUDA 12.1 support #16557

ddelange opened this issue May 10, 2024 · 4 comments · May be fixed by #16560
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@ddelange
Copy link
Contributor

ddelange commented May 10, 2024 •
edited

/kind feature

1. Describe IN DETAIL the feature/behavior/change you would like to see.

Bump the nvidia driver for CUDA 12.1 support.

We are currently running our 1.26 cluster configured with DriverPackage nvidia-driver-535 (with CUDA 12.0 support).

Note that we moved away from nvidia-headless-XXX-server because it does not install nvidia-smi and some other binaries on the host system, which caused issues for us with some cuda docker images that rely on the host having them available. It's barely any additional disk space used, so that was a quick and effective fix.

When we tried bumping to nvidia-driver-550 on EC2, the nodes stopped registering in the cluster. It was reverted before I could pull logs from the host system I'm afraid. I hope that your CI catches this.

2. Feel free to provide a design supporting your feature request.
@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label May 10, 2024
@ddelange
Copy link
Contributor Author

cc @hakman via blame:)

@hakman
Copy link
Member

hakman commented May 10, 2024

Unfortunately, our CI does not run on GPU nodes. We don't have credits to run such tests at the moment...

@ddelange
Copy link
Contributor Author

ddelange commented May 10, 2024 •
edited

just double-checked: nvidia-driver-535 already supports CUDA 12.1, which runs stable in our cluster

@ddelange ddelange linked a pull request May 10, 2024 that will close this issue
Open
@ddelange
Copy link
Contributor Author

Opened #16560

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Bump nvidia-driver-535 for CUDA 12 support ddelange/kops
3 participants
@hakman @ddelange @k8s-ci-robot