Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: don't try to clean up pvs on nodes that are gone #480

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

marcusramberg
Copy link

We're running local-provisioner to provide local storage for CI runners where nodes come and go pretty frequently.
We observe that the provisioner is trying run clean up on nodes that are already gone, which causes helper pods
to be stuck in pending state as they cannot be scheduled.

This PR adds a check to see if the node still exists before trying to clean up the node.

@marcusramberg marcusramberg changed the title fix: don't try to clean up pvcs on nodes that are gone fix: don't try to clean up pvs on nodes that are gone Feb 27, 2025
@marcusramberg marcusramberg force-pushed the marcus/ephemeral_fix branch 4 times, most recently from 69c6989 to 1e1388b Compare March 6, 2025 09:07
@marcusramberg
Copy link
Author

@derekbit Thoughts about this PR? We're running it in production from a fork now and it has resolved our issue of stuck pvs from old nodes and stuck helper prs trying to schedule on non-existing nodes. I guess it would also address the issues you're seeing in #416 with stuck pvs from previous runs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant