[Flaky Tests] Update/revert kindest/node to v1.32.0

[LucaBernstein]

How to categorize this PR?

/area testing
/kind flake

What this PR does / why we need it:
Fix the flakes introduced with #11510.
Due to a racy locking behavior in go-cni, which has been included in containerd v2.0.2, upon the cni restart, our tests can get stuck and flake.

The image kindest/node v1.32.0 used in this PR is based on kind v0.25.0 includes both still-supported versions for runc (v1.1.13) and containerd (v1.7.18).

Which issue(s) this PR fixes:
Flakes:

• https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/11540/pull-gardener-e2e-kind-upgrade/1895145575108579328#1:build-log.txt%3A541
• https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/11085/pull-gardener-e2e-kind-gardenadm/1895113419741204480#1:build-log.txt%3A388
• https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/11550/pull-gardener-e2e-kind-upgrade/1895481913641013248#1:build-log.txt%3A540
• https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/11550/pull-gardener-e2e-kind-gardenadm/1895481912210755584#1:build-log.txt%3A387

Special notes for your reviewer:
/cc @ScheererJ

Release note:

NONE

[marc1404]

🌧️ e2e-kind-upgrade failed with:

 - garden:statefulset/etcd: Unschedulable: 0/1 nodes available: 1 node is not ready
    - garden:pod/etcd-0: Unschedulable: 0/1 nodes available: 1 node is not ready
1/1 deployment(s) failed

[LucaBernstein]

/test all

[ScheererJ]

/assign

[LucaBernstein]

/test all

[LucaBernstein]

pull-gardener-e2e-kind-migration-ha-single-zone fails because of gardener/etcd-backup-restore#847.

[marc1404]

Great find! 😮 🦅

Just asking to be sure: Do we need to apply the containerd override for the cluster node(s) spawned through kind-up.sh as well?

gardener/hack/kind-up.sh

Lines 364 to 384 in 934b267

	# workarounds for KinD issues
	# TODO(marc1404): Remove once kindest/node uses runc >= v1.2.4
	if [[ -n "${CI:-}" ]]; then
	cp /get-runc/runc "$(dirname "$0")/../pkg/provider-local/node/runc"
	else
	"$(dirname "$0")/../pkg/provider-local/node/get-runc.sh"
	fi
	for node in $nodes; do
	# workaround https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files
	docker exec "$node" sh -c "sysctl fs.inotify.max_user_instances=8192"
	# TODO(marc1404): Remove once kindest/node uses runc >= v1.2.4
	# workaround issue with runc v1.2.3 provided by kindest/node:v1.32.0 by installing runc v1.2.4 manually (https://github.com/opencontainers/runc/pull/4555)
	if [ -n "${CI:-}" ]; then
	echo "Installing runc on node $node from container filesystem"
	docker cp /get-runc/runc "$node":/usr/local/sbin/runc
	else
	docker cp "$(dirname "$0")/../pkg/provider-local/node/runc" "$node":/usr/local/sbin/runc
	fi
	done

As I understand it the issue with go-cni seems to affect our tests primarily.

[LucaBernstein]

/test all
to increase certainty in stability

[LucaBernstein]

Just asking to be sure: Do we need to apply the containerd override for the cluster node(s) spawned through kind-up.sh as well?

In that case we would need to pre-build the image, as it's only referenced there.
As it affects only local setups and tests, imho we could go with the current patch alone and wait for a updated release soon.

[rfranzke]

In this light, do we need all this at all, i.e., can't we go back v1.31 and wait for a proper patch for v1.32 to be released, before we upgrade to it again? This way, we don't need to introduce these workarounds? Or is there something in v1.32 that we desperately need now?

[LucaBernstein]

Also addresse @marc1404's concern and override containerd in kind node, as local failure(s) have also been observed there by now.

In this light, do we need all this at all, i.e., can't we go back v1.31 and wait for a proper patch for v1.32 to be released, before we upgrade to it again? This way, we don't need to introduce these workarounds? Or is there something in v1.32 that we desperately need now?

The proposed changes make the problem + fix more explicit. I personally would vote for this "forward-fix".

[rfranzke]

I'm a fan of "fix-forward" if there is a real problem :) However, if there is nothing we gain out of v1.32 right now, I don't quite understand why we spend all this time to fix a problem that we wouldn't had if we just waited for the next release. Now we have workarounds and TODOs in our code and gain basically nothing out of it. 😕
Given that it's too late now, I'm OK with merging this PR, of course...

[LucaBernstein]

After another round of discussion we concluded that it might be best to revert/update to kindest/node v1.32.0 and get rid of both the runc and containerd version overrides.

[LucaBernstein]

working as intended:

runc version 1.1.13
commit: v1.1.13-0-g58aa920
spec: 1.0.2-dev
go: go1.22.9
libseccomp: 2.5.4
containerd github.com/containerd/containerd v1.7.18 ae71819c4f5e67bb4d5ae76a6b735f29cc25774e

[rfranzke]

/approve

[marc1404]

/lgtm
/approve

Thank you! After a lot of back and forth I'm glad we can remove this workaround again 😅

I've added a commit to, temporarily, require a Renovate dashboard approval for patch updates of kindest/node. I'll take care of checking future kindest/node releases on whether they ship with the correct version of runc and containerd in order to remove this restriction again.
2221f70 (#11532)

[gardener-prow]

LGTM label has been added.

Git tree hash: dd7c93a89e65a457eebe3fd4692dbe7f539f2d82

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: marc1404, rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [marc1404,rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment