Document VPA behavior for sidecar containers and how to change it

[voelzmo]

Which component are you using?:
/area vertical-pod-autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

The VPA is designed to ignore injected sidecar containers (see #5617 for context on why this decision was taken). This works properly, if the admission-controller webhook is executed before all sidecar injecting webhooks, which means its name has to come alphabetically before any sidecar injecting webhooks.
This is an implementation detail and not really an API, but that's how it currently works.

Describe the solution you'd like.:

• describe that VPA ignores sidecars and explain why it does so by default
  • default VPA containerPolicies include all Containers in a Pod, as soon as you have a VPA for a Pod. Even if you only have specific containerPolicy for a single Container, all other Containers will get the default setting, which contains .containerPolicy.mode: auto
  • if, by coincidence, a webhook which injects sidecars comes after the VPA webhook, it may be the case that it also sets resources for this injected Container, overwriting the resources that the VPA admission-controller just set. This would result in an endless eviction loop.
  • solution was to add an annotation to the Pod listing all Containers that were there "originally", before any webhooks injected anything. VPA only acts on Containers in this list. This annotation is added in the VPA admission-controller webhook, therefore this mechanism only works, if it comes before all sidecar injecting webhooks
• describe what you can do if you want injected sidecars to be managed by VPA
  • make VPA webhook come after all injecting webhooks
  • set reinvocationPolicy: IfNeeded on the VPA webhook, in case you have sidecar injecting webhooks which modify resources