Handling of inconsistent sidecar status in workspace

[triffer]

Description
The origin of this is the Operational Awarness workshop item #11.b. The assumption is, that if there is sidecar injection enabled or disabled in a namespace, all pods in the namespace should have a sidecar or no sidecar.
There is the scenario where the Injection-label is changed or removed while workloads were already deployed. This might result in a communication issue between workloads.
It should also be taken into account that the injection can be controlled by a label at pod level.

The proposed mitigation options were:

• Changing/removing the injection-label is reverted by the Istio operator
• Monitoring is established to detect whether the injection-label is consistently reflected in all deployed pods (=> all have a sidecar or all don't have a sidecar configured)

Reasons
The state of the sidecars in the namespace should be consistent with the configurations of the injection of the namespace. A customer might not be aware that changing this configuration requires a restart of the already deployed pods.

DoD:

• provide documentation
• release notes and What's New updates for Kyma customers
• provide unit tests
• provide integration tests
• test on production-like environment
• verify resource limits
• followup issue
• PR reviewer will verify code coverage and evaluate if it is acceptable

Attachments

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[kyma-bot]

This issue or PR has been automatically closed due to the lack of activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle stale

If you think that I work incorrectly, kindly raise an issue with the problem.

/close

[kyma-bot]

@kyma-bot: Closing this issue.

In response to this:

This issue or PR has been automatically closed due to the lack of activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle stale

If you think that I work incorrectly, kindly raise an issue with the problem.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale