KEP 1618: Optional garbage collection of finished Workloads #2742

mwysokin · 2024-08-01T11:34:57Z

What type of PR is this?

/kind documentation
/kind api-change

What this PR does / why we need it:

Which issue(s) this PR fixes:

KEP for #1618

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

k8s-ci-robot · 2024-08-01T11:35:06Z

Hi @mwysokin. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2024-08-01T11:35:14Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`2efe734`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/66b60ece996fd600085cf14a

mimowo · 2024-08-01T11:40:07Z

/cc @tenzen-y @PBundyra

mimowo · 2024-08-01T11:50:10Z

nit, please say "KEP for #1618" in the PR description instead of "Fixes". Otherwise the robot will close the issue after merging the PR :)

mwysokin · 2024-08-01T11:56:06Z

/assign

mimowo

First pass, it lgtm overall. Left some comments to improve it.

keps/1618-optional-gc-of-workloads/README.md

mimowo · 2024-08-02T08:06:04Z

/ok-to-test

…ories into 1 and move the description of the behavior to a new section under Design Details.

keps/1618-optional-gc-of-workloads/README.md

mimowo · 2024-08-08T07:21:50Z

keps/1618-optional-gc-of-workloads/README.md

+leading to unnecessary use of etcd storage and a gradual increase in Kueue's memory footprint. Some of these objects, 
+like finished Workloads, do not contain any useful information that could be used for additional purposes, such as debugging. 
+That's why, based on user feedback in [#1618](https://github.com/kubernetes-sigs/kueue/issues/1618)
+or [#1789](https://github.com/kubernetes-sigs/kueue/issues/1789), a mechanism for garbage collection of finished Workloads 


QQ: I'm wondering if the KEP and the initial implementation tackle this issue, because the workloads in the context of this issue are not necessarily finished. It might still be ok to mention in motivation for the generic KEP on workload cleanups, but I would like to clarify if the first version will cover it.

I checked and even though an extension to the code would be minimal I'd rather keep it as a separate task not to pollute and further delay this feature. I will update the KEP once I have something more polished then just a draft version of how the removal of orphaned workloads would look like. For now I'm going to remove the orphaned Workloads issue from the KEP. WDYT?

Thank you for checking I think removing them from the KEP makes sense, because:

they are not finished

there is no point in keeping such workloads for >30min (or whatever is configured).
So, I consider Workloads corresponding to Jobs deleted with --cascede=orphan still consume resources #1789 to be just a bug which does not require any API changes or KEP,.

Actually, on the second thought. I think it is better to honor user's choice of using "orphan" mode. So, I would suggest to just transition them to finished state. This way we could again link the issue and the KEP.

mimowo

LGTM, one remaining question from me: https://github.com/kubernetes-sigs/kueue/pull/2742/files#r1708818068. @tenzen-y would you like to give it a pass?

mimowo

/lgtm
/approve
/hold
To wait for feedback from @tenzen-y as another pair of eyes.

k8s-ci-robot · 2024-08-08T11:53:15Z

LGTM label has been added.

Git tree hash: 4b4f2f1108238ec7b0edef46b807fe0cc471a4d8

k8s-ci-robot · 2024-08-08T11:53:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mimowo, mwysokin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

PBundyra · 2024-08-08T12:33:03Z

LGTM from my side as well

tenzen-y · 2024-08-08T14:59:52Z

/lgtm /approve /hold To wait for feedback from @tenzen-y as another pair of eyes.

ACK

tenzen-y · 2024-08-08T15:00:29Z

/remove-kind api-change

tenzen-y · 2024-08-08T15:09:43Z

keps/1618-optional-gc-of-workloads/README.md

+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.


Does this mean that you will work on the deletion of other objects except the finished Workloads in the next iteration?

But, you mention that ''Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.'' in the Non-Goals section.

The goal was to include certain extensibility into the design so that other objects could be added without modifying root lvl API (Jobs, etc.). That's the main reason why the API key for Workloads is nested. I haven't seen any issues for removal of other completed object (other than orphaned Workloads #1789) but if there's any interest in extending that feature I'd be happy to work on it but figuring out how to remove every object that Kueue owns wasn't the goal here, that's where the items in goals/non-goals came from. I'm also happy to work on any updates to this KEP if we decide to extend the cleanup process to other objects.

I would recommend limiting the scope for this KEP to Kueue managed objects, not only to Kueue Workload objects since actually, we aim to extend the API for the future extensibility for the other Kueue managed resources like AdmissionCheck.

The v1beta1 means that we can not bring breaking changes to APIs without bumping the API version.
So, I would like to take care of future API possibilities in this KEP.

So, my recommendation is to remove "starting with finished Workloads." from Goal, and move the statement to the proposal. And then, mention that we do not aim to add APIs for the Jobs like batch/job, RayJob as we discussed in another thread...
I'm wondering if we need to add another mechanisms for the Jobs retention in the future when we find the actual use cases.

Got it. Let me propose another set of changes before the end of the week.

Introduce configuration of global retention policies for Kueue-authored Kubernetes objects

This sentence matches Workload and ProvisioiningRequest. We don't need to clean up provisioning requests explicitly, because they are children of the Workloads.

So, for now, we only need to take care of Workloads. But I agree we should keep the API extensible for supporting the cleanup of finished jobs, but that can be addressed in a separate KEP.

tenzen-y · 2024-08-08T15:20:36Z

keps/1618-optional-gc-of-workloads/README.md

+- **R**: In clusters with a large number of existing finished Workloads 
+(thousands, tens of thousands, etc.), it may take a significant amount of time
+to delete all previously finished Workloads that qualify for deletion. Since the 
+reconciliation loop is effectively synchronous, this may impact the reconciliation 
+of new jobs. From the user's perspective, it may seem like Kueue's initialization 
+time is very long for that initial run until all expired objects are deleted.\
+**M**: Unfortunately, there is no easy way to mitigate this without complicating the code.
+A potential improvement would be to have a dedicated internal queue that handles deletion 
+outside the reconciliation loop, or to delete expired objects in batches so that after 
+a batch of deletions is completed, any remaining expired objects would be skipped until 
+the next run of the loop. However, this would go against the synchronous nature of the reconciliation loop.


I believe that we can mitigate this risk by setting the strictly qps only when admins enable this feature.
Once we limit the client qps and burst in the KueueConfiguration, we can mitigate the burst kube-apiserver load.

kueue/config/components/manager/controller_manager_config.yaml

Lines 21 to 23 in c6f181b

clientConnection:

qps: 50

burst: 100

So, I would propose the mentioning limiting client qps when the admins enable this feature in the existing cluster.
Hence, could you mention this mitigation here?

tenzen-y · 2024-08-08T15:33:37Z

keps/1618-optional-gc-of-workloads/README.md

+-->
+
+
+### API


TBH, I feel that this API looks like a half-bake to me.
Because when we add new policies to garbage collect other resources like integrations Jobs, we need to hard cord the API name here like FinishedRayClusterRetention. But, the Kueue integration Jobs are the extensible interface for external Jobs like AppWrapper or in-house one. So, I guess that it is challenging to maintain retention policies for all integrations like:

[...] objectRetentionPolicies: finiishedWorkloadRetention: 10s failedRayClusterRetention: 30s # How to specify in-house or other OSS resource policy, here? # In the upstream Kueue, we can not find all integrations # since the integration framework could be implemented in other places regardless of private or public repositories. [...]

From another perspective, even if we restrict the objective objects to specify retention duration, this API is a little bit extensible because in spite of that the API says that this API is used for retention policies for "object", but in this case, actually, we can specify only Workload retention policies.

objectRetentionPolicies: finishedWorkloadRetention: 10s succeededWorkloadRetention: 30s

In conclusion, I would ask which objects will you support in this feature through all iterations?
It's only Workload? or It's arbitrary objects?

TBH I started working on this KEP without too much vision but rather I wanted to implement #1618 and I wanted to make the API flexible enough so that it could be extended with other object types in the future. After reading what you said it indeed makes sense to come up with a consistent approach for all objects so that user's could benefit from Kueue's extensibility.

I don't have much experience with KEPs so I'm not sure what should happen now. For sure I'd like to ship the feature so that it doesn't slowly fades away in the in-review stage but at the same time if you feel that it's half-baked maybe I need to make a step back and come up with a more flexible approach. All feedback and guidance is really welcome 🙇

I'll do some research over the weekend and I'll try to propose something early next week.

There are a couple of questions / aspects I see touched here:

should we make it generic to cover Job CRDs?

should we nest the API inside objectRetentionPolicies?

Re 1: There is an intrinsic asymmetry between workloads and Job CRDs. Job CRDs are created by users who can specify spec.ttlSecondsAfterFinished. Workloads are created by Kueue, and so users don't have this control. So, I think it is justified to start from Workloads only and have the API field specific to them.

Another API maintained by Kueue that comes to my mind is ProvisioningRequest. I think having finishedProvisioningRequestRetention will make sense too (but we can leave it for later as currently we delete them eagerly anyway).

Re 2: I think it makes sense to nest the API inside objectRetentionPolicies to be future-proof in case we have some other objects with lifetime maintained by Kueue (like provisioning requests, or different policies for finished workloads depending on success /failure).

Re 1: There is an intrinsic asymmetry between workloads and Job CRDs. Job CRDs are created by users who can specify spec.ttlSecondsAfterFinished. Workloads are created by Kueue, and so users don't have this control. So, I think it is justified to start from Workloads only and have the API field specific to them.

Yes, I agree with you. TBH, I don't prefer to implement the feature to do garbage collection of Jobs since I'm thinking that deletion jobs are responsible for Job's own controller like ttlSecondsAfterFinished. But this proposed API is extensible for Job API. So, I asked, "which objects will you support in this feature".

Another API maintained by Kueue that comes to my mind is ProvisioningRequest. I think having finishedProvisioningRequestRetention will make sense too (but we can leave it for later as currently we delete them eagerly anyway).

I agree with you. I believe that the separate and dedicated CRDs for this feature are overkill. So, I would recommend not introducing the dedicated CRD.

Re 2: I think it makes sense to nest the API inside objectRetentionPolicies to be future-proof in case we have some other objects with lifetime maintained by Kueue (like provisioning requests, or different policies for finished workloads depending on success /failure).

If we want to support additional Kueue-managed objects in the future,
could we add a dedicated field for each object something like this?
I guess that the criterion for garbage collection depends on the object’s kind.

workloadRetentionPolicy: onConditions: [metav1.Condition] admissionCheckRetentionsPolicy: onConditions: [metav1.Condition] onResourceFlavor: name: xxx ...

This looks a bit more complex than I originally anticipated but I agree that it's quite extensible. TBH I lack expertise when it comes evaluating whether this would be a consistent and a good way for Kueue to support that so I'll defer to other maintainers (@mimowo, @alculquicondor). Once we reach the consensus I'll be happy to implement it.

I prefer 2. It looks more future-proof.

That sounds good to me.

This is a side note, although. To evolve API more easily, I want to find a way to guard the new API field :)

IIUC, k/k has the feature gate guard for the new API fields like here: https://github.com/kubernetes/kubernetes/blob/1c4f540669397d031d464877ec3d2b8ac6be9b2f/pkg/apis/core/types.go#L543-L548

We can not break the new API field without bumping the API version in the current API evolving steps. I think that a new API without a guard makes API evolution harder since we need to consider future compatibility and extensibility in the first implementation.

The feature gate only implies that the field will be ignored when disabled. It doesn't mean that we can change the API in a future version of kubernetes, without changing the API version.

Oh, I see. That was not my expected behavior.
Thanks for letting me know.

tenzen-y · 2024-08-08T15:44:02Z

keps/1618-optional-gc-of-workloads/README.md

+[//]: # (### Graduation Criteria)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # ()
+[//]: # (Clearly define what it means for the feature to be implemented and)
+
+[//]: # (considered stable.)
+
+[//]: # ()
+[//]: # (If the feature you are introducing has high complexity, consider adding graduation)
+
+[//]: # (milestones with these graduation criteria:)
+
+[//]: # (- [Maturity levels &#40;`alpha`, `beta`, `stable`&#41;][maturity-levels])
+
+[//]: # (- [Feature gate][feature gate] lifecycle)
+
+[//]: # (- [Deprecation policy][deprecation-policy])
+
+[//]: # ()
+[//]: # ([feature gate]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md)
+
+[//]: # ([maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions)
+
+[//]: # ([deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/)
+
+[//]: # (-->)


Could you revert these comment-outs since we need to revisit here when we want to graduate this feature to Beta/Stable?

I explicitly commented it out instead of removing it so it could be revisited later but maybe it needs to be there from the beginning? It's my first KEP ever so I'm learning as I go and I'll follow your guidance 🙏

k8s-ci-robot · 2024-08-09T12:50:11Z

@mwysokin: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kueue-verify-main	`2efe734`	link	true	`/test pull-kueue-verify-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

alculquicondor · 2024-08-13T17:28:25Z

/assign

alculquicondor · 2024-08-16T15:19:49Z

keps/1618-optional-gc-of-workloads/README.md

+
+- Support the deletion of finished Workload objects.
+- Introduce configuration of global retention policies for Kueue-authored Kubernetes objects, 
+starting with finished Workloads.


Introduce configuration of global retention policies for Kueue-authored Kubernetes objects

This sentence matches Workload and ProvisioiningRequest. We don't need to clean up provisioning requests explicitly, because they are children of the Workloads.

So, for now, we only need to take care of Workloads. But I agree we should keep the API extensible for supporting the cleanup of finished jobs, but that can be addressed in a separate KEP.

alculquicondor · 2024-08-16T15:20:23Z

keps/1618-optional-gc-of-workloads/README.md

+and make progress.
+-->
+
+- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.


Suggested change

- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.

- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.

alculquicondor · 2024-08-16T15:20:56Z

keps/1618-optional-gc-of-workloads/README.md

+
+## Summary
+
+<!--


Please remove the comments

alculquicondor · 2024-08-16T15:21:57Z

keps/1618-optional-gc-of-workloads/README.md

+nitty-gritty.
+-->
+
+Add a new API called `objectRetentionPolicies`, 


Suggested change

Add a new API called `objectRetentionPolicies`,

Add a new field called `objectRetentionPolicies` to the Kueue Configuration API,

alculquicondor · 2024-08-16T15:22:09Z

keps/1618-optional-gc-of-workloads/README.md

+Add a new API called `objectRetentionPolicies`, 
+which will enable specifying a retention policy for finished Workloads under an option 
+called `FinishedWorkloadRetention`. This API section could be extended in the future 
+with options for other Kueue-authored objects.


Suggested change

with options for other Kueue-authored objects.

with options for other Kueue-managed objects.

alculquicondor · 2024-08-16T15:24:01Z

keps/1618-optional-gc-of-workloads/README.md

+-->
+
+- Initially, other naming was considered for config keys. Namely, instead of "Objects," 
+the word "Resources" was used, but based on feedback from `@alculquicondor`, it was changed 


we don't usually keep the names of the people who suggested the changes.

alculquicondor · 2024-08-16T15:32:59Z

keps/1618-optional-gc-of-workloads/README.md

+-->
+
+
+### API


I think that API looks too complex. For the most part, we will be happy just cleaning up finished Workloads.

Let me propose the following:

objectRetentionPolicies: workloads: afterFinished: 10m

It leaves us the possibility to extend to other objects and to even add conditions, but I prefer we don't give that flexibility yet.

alculquicondor · 2024-08-16T15:34:21Z

keps/1618-optional-gc-of-workloads/README.md

+[//]: # (##### Prerequisite testing updates)
+
+[//]: # ()
+[//]: # (<!--)
+
+[//]: # (Based on reviewers feedback describe what additional tests need to be added prior)
+
+[//]: # (implementing this enhancement to ensure the enhancements have also solid foundations.)
+
+[//]: # (-->)


You can remove this if there aren't any requirements.

mwysokin · 2024-08-26T05:33:18Z

FYI I just wanted to let everyone know that I didn't abandon the work. I've been on vacation last week and I'll be back in September. Sorry for the delay 🙇

tenzen-y · 2024-08-26T22:52:15Z

FYI I just wanted to let everyone know that I didn't abandon the work. I've been on vacation last week and I'll be back in September. Sorry for the delay 🙇

No worries. I'm looking forward to moving this discussion after you come back here!

mwysokin · 2024-09-27T08:59:09Z

I'm really sorry it's been dragging. I'm coming back to it starting Monday.

mwysokin added 2 commits July 31, 2024 17:29

Initial commit 95% done

19dfa94

KEP ready to be reviewed

334b020

k8s-ci-robot requested a review from denkensk August 1, 2024 11:35

k8s-ci-robot requested a review from mimowo August 1, 2024 11:35

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 1, 2024

k8s-ci-robot requested review from PBundyra and tenzen-y August 1, 2024 11:40

mwysokin added 2 commits August 1, 2024 11:44

Fix TOC

dfa5421

Update reviewers

1ddf7f5

Fix API formatting

482652e

k8s-ci-robot assigned mwysokin Aug 1, 2024

mimowo reviewed Aug 1, 2024

View reviewed changes

keps/1618-optional-gc-of-workloads/README.md Outdated Show resolved Hide resolved

keps/1618-optional-gc-of-workloads/README.md Outdated Show resolved Hide resolved

keps/1618-optional-gc-of-workloads/README.md Show resolved Hide resolved

PBundyra reviewed Aug 1, 2024

View reviewed changes

keps/1618-optional-gc-of-workloads/README.md Outdated Show resolved Hide resolved

PBundyra reviewed Aug 1, 2024

View reviewed changes

keps/1618-optional-gc-of-workloads/README.md Outdated Show resolved Hide resolved

PBundyra reviewed Aug 1, 2024

View reviewed changes

keps/1618-optional-gc-of-workloads/README.md Show resolved Hide resolved

PBundyra reviewed Aug 1, 2024

View reviewed changes

keps/1618-optional-gc-of-workloads/README.md Show resolved Hide resolved

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 2, 2024

REVIEW: Move misconfiguration and default scenario to notes. Merge st…

344a512

…ories into 1 and move the description of the behavior to a new section under Design Details.

Update TOC

762ac09

mimowo reviewed Aug 7, 2024

View reviewed changes

keps/1618-optional-gc-of-workloads/README.md Outdated Show resolved Hide resolved

mwysokin added 2 commits August 7, 2024 15:39

Drop redundant comment

43a36f3

REVIEW: Move an alternative approach from Notes to Alternatives

2d59922

mimowo reviewed Aug 8, 2024

View reviewed changes

mwysokin added 2 commits August 8, 2024 13:20

REVIEW: Move orphanes workloads mention to the notes

2f97fdf

REVIEW: Remove mention of orphaned workloads

a43df30

mimowo reviewed Aug 8, 2024

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 8, 2024

k8s-ci-robot assigned mimowo Aug 8, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 8, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 8, 2024

k8s-ci-robot removed the kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API label Aug 8, 2024

tenzen-y reviewed Aug 8, 2024

View reviewed changes

REVIEW: Bring back Graduation Criteria section

2efe734

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 9, 2024

k8s-ci-robot requested a review from mimowo August 9, 2024 12:42

k8s-ci-robot assigned alculquicondor Aug 13, 2024

alculquicondor reviewed Aug 16, 2024

View reviewed changes

	- Support the deletion/expiration of any Kueue-authored Kubernetes object other than finished Workloads.
	- Support the deletion/expiration of jobs or other Kubernetes objects not authored by Kueue.

	Add a new API called `objectRetentionPolicies`,
	Add a new field called `objectRetentionPolicies` to the Kueue Configuration API,

	with options for other Kueue-authored objects.
	with options for other Kueue-managed objects.

		-->


		### API


		## Summary

		<!--

		-->


		### API

KEP 1618: Optional garbage collection of finished Workloads #2742

Are you sure you want to change the base?

KEP 1618: Optional garbage collection of finished Workloads #2742

Conversation

mwysokin commented Aug 1, 2024 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Aug 1, 2024

netlify bot commented Aug 1, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

mimowo commented Aug 1, 2024

mimowo commented Aug 1, 2024

mwysokin commented Aug 1, 2024

mimowo left a comment

Choose a reason for hiding this comment

mimowo commented Aug 2, 2024

mimowo Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

mwysokin Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mimowo left a comment

Choose a reason for hiding this comment

mimowo left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 8, 2024

k8s-ci-robot commented Aug 8, 2024

PBundyra commented Aug 8, 2024

tenzen-y commented Aug 8, 2024

tenzen-y commented Aug 8, 2024

Choose a reason for hiding this comment

mwysokin Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwysokin Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 9, 2024

alculquicondor commented Aug 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwysokin commented Aug 26, 2024

tenzen-y commented Aug 26, 2024

mwysokin commented Sep 27, 2024

mwysokin commented Aug 1, 2024 •

edited

Loading

netlify bot commented Aug 1, 2024 •

edited

Loading

mimowo Aug 8, 2024 •

edited

Loading

mwysokin Aug 8, 2024 •

edited

Loading

mwysokin Aug 9, 2024 •

edited

Loading

mwysokin Aug 14, 2024 •

edited

Loading

alculquicondor Aug 16, 2024 •

edited

Loading