Change recommendation on sidecars to use new kube sidecars feature

[mikekap]

There is a new sidecar feature that puts sidecars into initContainers with restartPolicy: Always. It is on by default in kube 1.29+.

The docs should at least encourage people to use it. Otherwise, it might make sense to move the agent and checkout containers there so the job pods created by agent-stack-k8s don't "look weird" (i.e. they are NotReady after checkout finishes since one of the containers shuts down).

[DrJosh9000]

It's a good suggestion @mikekap.

I'm going to leave some notes here for the benefit of the people who have recently come aboard as maintainers:

I did a little investigating last year into making all the containers but the last "command" container an init container, including making the agent container a "real" sidecar (an init container with restartPolicy:Always): https://github.com/buildkite/agent-stack-k8s/tree/oops-all-init-containers

Two benefits of this approach are

• the container sequencing logic in buildkite-agent can be deleted, since all the other containers will run sequentially when appropriately arranged
• pod resources should be lower and more efficient, since there won't be 3 or more containers running "in parallel" where really only 2 are doing anything useful at any given moment.

A blocker will be the number of users on K8s versions lower than 1.29. If the stack could detect the SidecarContainers feature gate, then it could conditionally use such an approach, but that would make internal/controller/scheduler/scheduler.go even bigger.

[mikekap]

All fair points. To some extent this issue is to at least update the docs to encourage folks to use initContainer-based sidecars.

One small note is that technically kube 1.28 reached EOL in October 2024 - so all “supported” versions of kube have this feature. Granted, we just upgraded from kube 1.24 a month ago so it’s a good point that older kube cluster support isn’t easily black-and-white.