Skip to content

[Bug] stale resource not cleaned up #2351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
1 task done
nandiheath opened this issue Apr 16, 2025 · 0 comments · May be fixed by #2353
Open
1 task done

[Bug] stale resource not cleaned up #2351

nandiheath opened this issue Apr 16, 2025 · 0 comments · May be fixed by #2353
Labels
bug Something isn't working

Comments

@nandiheath
Copy link

chainsaw version Version

v0.2.4

Description

We're running some repetitive tests that reuse the same resource name. Normally, the resource is cleaned up once the test completes.However if the test exits unexpectedly, the cleanup may not occur, leaving the resource behind. When this happens, subsequent test runs don't attempt to clean up the existing resource, resulting in it being left stale indefinitely.

I’m wondering if this is because the current cleanup logic only considers resources that were explicitly created during the test run.

Steps to reproduce

  1. Start a test that creates a PVC and a Job which mounts it.
  2. Interrupt the test after the PVC is created.
  3. Run the test again — the previously created PVC will now be left stale after the test completes.

Expected behavior

All resources should be cleaned up after the test.

Screenshots

No response

Logs

The log for the second test showing the PVC is `PATCH` instead of `CREATE`

    | 09:00:28 | ebs-csi-driver | ephemeral storageClass     | TRY       | BEGIN |
    | 09:00:28 | ebs-csi-driver | ephemeral storageClass     | APPLY     | RUN   | storage.k8s.io/v1/StorageClass @ chainsaw-ebs-csi-driver-kuberhealthy
    | 09:00:29 | ebs-csi-driver | ephemeral storageClass     | PATCH     | OK    | storage.k8s.io/v1/StorageClass @ chainsaw-ebs-csi-driver-kuberhealthy
    | 09:00:29 | ebs-csi-driver | ephemeral storageClass     | APPLY     | DONE  | storage.k8s.io/v1/StorageClass @ chainsaw-ebs-csi-driver-kuberhealthy
    | 09:00:29 | ebs-csi-driver | ephemeral storageClass     | TRY       | END   |
    | 09:00:29 | ebs-csi-driver | persistent volume claim    | TRY       | BEGIN |
    | 09:00:29 | ebs-csi-driver | persistent volume claim    | APPLY     | RUN   | v1/PersistentVolumeClaim @ kube-system/chainsaw-ebs-csi-driver-kuberhealthy
    | 09:00:29 | ebs-csi-driver | persistent volume claim    | PATCH     | OK    | v1/PersistentVolumeClaim @ kube-system/chainsaw-ebs-csi-driver-kuberhealthy
    | 09:00:29 | ebs-csi-driver | persistent volume claim    | APPLY     | DONE  | v1/PersistentVolumeClaim @ kube-system/chainsaw-ebs-csi-driver-kuberhealthy
    | 09:00:29 | ebs-csi-driver | persistent volume claim    | CMD       | RUN   |

Slack discussion

No response

Troubleshooting

  • I have searched other issues in this repository and mine is not recorded.
@nandiheath nandiheath added the bug Something isn't working label Apr 16, 2025
ash01ish added a commit to ash01ish/chainsaw that referenced this issue Apr 17, 2025
@ash01ish
fix: handle existing resources gracefully in create operation kyverno…
ff40554 
…#2351

When a test exits unexpectedly after creating resources, those resources
aren't cleaned up. When the test is run again, it detects the existing
resources but returns an error instead of cleaning them up, leaving them
stale indefinitely.

This change modifies the tryCreateResource method to not return an error
when a resource already exists, but instead continue with the operation.
This ensures that pre-existing resources are properly cleaned up at the
end of the test.

Fixes kyverno#2351
@ash01ish ash01ish linked a pull request Apr 17, 2025 that will close this issue
Open
ash01ish added a commit to ash01ish/chainsaw that referenced this issue Apr 17, 2025
@ash01ish
fix: handle existing resources gracefully in create operation kyverno…
51cc5af 
…#2351

When a test exits unexpectedly after creating resources, those resources
aren't cleaned up. When the test is run again, it detects the existing
resources but returns an error instead of cleaning them up, leaving them
stale indefinitely.

This change modifies the tryCreateResource method to not return an error
when a resource already exists, but instead continue with the operation.
This ensures that pre-existing resources are properly cleaned up at the
end of the test.

Fixes kyverno#2351

Signed-off-by: Ashish Thirunagari <[email protected]>
ash01ish added a commit to ash01ish/chainsaw that referenced this issue Apr 21, 2025
@ash01ish
fix: handle existing resources gracefully in create operation kyverno…
f7bc1a7 
…#2351

When a test exits unexpectedly after creating resources, those resources
aren't cleaned up. When the test is run again, it detects the existing
resources but returns an error instead of cleaning them up, leaving them
stale indefinitely.

This change modifies the tryCreateResource method to not return an error
when a resource already exists, but instead continue with the operation.
This ensures that pre-existing resources are properly cleaned up at the
end of the test.

Fixes kyverno#2351

Signed-off-by: Ashish Thirunagari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: handle existing resources gracefully in create operation #2351 ash01ish/chainsaw
1 participant
@nandiheath