Drop Kupid in favor of an alternative (OPA Gatekeeper or Kyverno...)

[timebertt]

What would you like to be added:

Readme says:

The OPA Gatekeeper allows to define policy to validate and mutate any kubernetes resource. Technically, this can be used to dynamically inject anything, including scheduling policy into pods. But this is too big a component to introduce just to dynamically inject scheduling policy. Besides, the policy definition as code is undesirable in this context because the policy itself would be non-declarative and hard to validate while deploying the policy.

However, it doesn't seem this justifies building our own component (which is currently unmaintained?) in comparison to the relatively low effort to reuse a well-established project from the community.

This repository could basically be a few yaml files instead of thousands of lines of code.

Why is this needed:

• relieve us from unnecessary maintenance effort (see open PRs, repository requires regular dependency updates, ref Support for k8s 1.22+ #32, see open dependabot vulnerability alerts)
• OPA Gatekeeper will open new doors for many other mechanisms (e.g. mutating specific shoot control planes)

[ashwani2k]

Thanks @timebertt for the suggestion. I'm game for it for same reasons.

If the switch is seemingly simple then we need to consider the following to achieve it.

1. Need to understand OPA semantics and the identify the steps to achieve the existing functionality of Kupid.
2. Identify the change required for removing Kupid from our landscapes and introduce OPA Gatekeeper.
3. Roll it out on productive landscape without disruption as currently etcd scheduling hinges on it.

However, If I understand we need to do it before we upgrade the landscapes to k8s 1.22.
So i'm not sure if we can avoid ref, #32 in time.

[ashwani2k]

I came across this https://kyverno.io/ in a conversation with @unmarshall. This looks exhaustive and might even be a good replacement for opa-gatekeeper
@timebertt, @dguendisch -- Do you think we will think of using kyverno instead of opa-gatekeeper in gardener landscapes for its declarative semantics which are similar to what kupid does although in a limited scope.

Just checking if we should relook?

[dguendisch]

kyverno looks straight forward, seems promising to me (though I have no experience with gatekeeper and kupid), maybe @hendrikKahl wants to comment as well.

[vlerenc]

Well, I have not a lot of practical experience, so probably the best colleagues to judge would be those already familiar with either (probably OPA Gatekeeper, then testing also Kyverno) but from what I have seen/heard/read, the answer is clearly Kyverno, because:

• OPA is more general (we do not have that requirement) while Kyverno specifically targets Kubernetes (only our environment) and its usage is therefore much more natural/native.
• Because of that, OPA requires you to learn a programming language (Rego), while you can write policies in Kyverno kustomize-style, which is much more approachable and we can assume a better internal adoption therefore.
• Kyverno is newer. Even though, it's docs and examples are much better (https://kyverno.io/policies).

[hendrikKahl]

Well, I used Gatekeeper (k8s implementation of OPA) to inject labels to specific pods. That worked quite well and could be done with a rather simple custom resource object. So I'm not too familiar with rego and wouldn't want to judge, whether it is suitable to replace the code in this repo.

After a first glimpse at kyverno, I think it is much easier to modify things or inject certain configuration. When it comes to policies, I guess OPA is still more powerful.

[vlerenc]

Yes, a programming language will always be more powerful. Question is, whether that's required or needless complexity. If it's only about Kubernetes, people seem to prefer and be just as expressive with Kyverno. Just saying - that seems to be the verdict of many who compared the two.

[unmarshall]

I don't have much experience with OPA/Rego, however when i looked at the OPA policy that was very recently created as a hack to make multi-zone seed, it was a not a very pretty site.
To have a look at it: connect to aws-ha seed and look at policies configmap k get cm -n opa policies -oyaml. Maybe it is not at all common to update/patch the configmap directly (as that would be very error prone). Try reading that minified rego and it will be fun ;)

[vlerenc]

Just read a bit more, indeed many recommend Kyverno in blogs and such.
See also: https://www.youtube.com/watch?v=9gSrRNmmKBc&t=926s and the following summary.
So, +1 for @unmarshall recommendation to look into/consider that over OPA Gatekeeper.

[ashwani2k]

Added a document #45 capturing the evaluation done on Kyverno as a replacement.
Looks promising to me.

cc @unmarshall @shreyas-s-rao

[timebertt]

I also recently tried out kyverno (just some basic stuff with validating and mutating policies).
It seemed very promising to me as well. It's very easy to setup and intuitive to use/understand :)