Enhance HA of ory-oathkeeper workloads

[ebensom]

Description

During chaos testing run, which simulated pod failures (kubelet) per AZ on all nodes belonging to the specific AZ, it was pointed out that critical ory-oathkeeper pods were all traniently down, either in pending state or in init state. The reason is that to all replicas were being scheduled in nodes belonging to the same AZ, and kept being do so during pod termination/eviction.

For ory-oathkeeper running on enterprise plan runtimes (having at least 3 nodes in 2 AZs), ensure that the pods are configured with either topology spread constraints, or pod anti affinity rules to preferably

• spread the replicas on different nodes, and furthermore
• spread the replicas on nodes belonging to different availability zones.

Reasons

Higher resiliency and availiability during node level and zones level failure scenarios.

DoD:

• Provide unit and integration tests.
• Provide documentation.
• Verify if the solution works for both open-source Kyma and SAP BTP, Kyma runtime.
• If you changed the resource limits, explain why it was needed.
• Verify that your contributions don't decrease code coverage. If they do, explain why this is the case.
• Add release notes.

Attachments

[ebensom]

@kyma-project/goat could you please prioritize this issue?