Filter manageJobsWithoutQueueNames using a namespaceSelector for all integrations

[dgrove-oss]

What would you like to be added:

Add a manageJobsWithoutQueueNameNamespaceSelector to the Kueue configuration API and use it to restrict the jobs to which manageJobsWithoutQueueName is applied across all integrations.

We can then deprecate podOptions.namepaceSelector and eventually remove it from the configuration API.

Why is this needed:

As discussed in the review of #3520, because the Deployment and StatefulSet integrations are built on top of the Pod integration, they implicitly restrict the scope of manageJobsWithoutQueueName by filtering it with podOptions.namepaceSelector. This results in an irregular API that may be confusing to users.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[dgrove-oss]

/assign

I will work on drafting a KEP.

[mimowo]

cc @mwielgus @tenzen-y @mwysokin

[tenzen-y]

IIRC, we did not introduce this mechanism in the previous discussions: #2119

Instead of introducing this knob, shouldn't we decouple the Stateful and Deployment integration from Pod integration?
Actually, the StatefulSet integration has dedicated controller, and I'm not sure the reason why we keep depending on the Pod integrations.

[dgrove-oss]

This was discussed before (#2119), but what changed since then is the addition of the Deployment and StatefulSet integrations.

Deployment and StatefulSet have the same fundamental issue with manageJobsWithoutQueueNames as Pod (independent of how we choose to implement the integrations). If manageJobsWithoutQueueNames is true and you don't restrict its scope somehow things will break because system namespaces that use Deployments or StatefulSets will unexpectedly be suspended, breaking the system's functionality.

So, the proposal is to put together a KEP to see what it would take to have a uniform API to modulate manageJobsWithoutQueueNames.

An alternative that we can discuss in the KEP is to instead deprecate and then remove manageJobsWithoutQueueNames entirely in favor of a ValidatingAdmissionPolicy. I've implemented such a policy as part of our MLBatch project (see admissionPolicy.yaml) and it was subtler than was suggested in the discussion of #2119. In particular, dealing with child Jobs requires teaching the VAP about ownership links and (at least for me) that was tricky and I'm not convinced it is less complex than the namespaceSelector option. Note that my VAP only deals with one level of ownership (because that's all I needed for our restricted use case). A general one would have to deal with crawling several links to get to the "top-level" Job that was being managed by Kueue.

[mimowo]

Regarding the Deployments and StatefulSet via pod-based integration.

The fundamental difference between Deployments, StatefulSet and JobCRDs is that there is no suspend field ( and it is unlikely to be ever added, or it will take a year at very least to get into beta).

So, using scheduling Gates for suspending seems like a good fit, and this is already provided by the pod-based integration, so we are reusing it. I'm open to alternatives, but I think they need to be presented more holistically, and in detail, because they seem vague at the moment to me at least.

[dgrove-oss]

Initial draft of the KEP in #3595