[WIP] Release Proposal v2.27

[tico88612]

(UPD: This proposal is WIP, need to re-organize some release notes.)

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  Change kubeadm_patches format to use an array of inline patch instead of patch files.
  See the example for new format. (Use in inventory variables rather than patch files for kubeadm_patches #11521, @VannTen)
• Action required
  Removes the generation of static tokens for every node in the cluster when kube_token_auth: true (Remove generation of static tokens for cluster members #11567, @VannTen)
• Action required
  The kubelet_node_{config_extra_args,custom_flags} are removed. Use kubelet_{config_extra_args,custom_flags} in <your_inventory>/group_vars/kube_node.yml.
  The {kube,system}_master_{cpu,memory,ephemeral-storage,pid} are removed. Use the {kube,system}_{cpu,memory,ephemeral-storage,pid} variables in <your_inventory>/group_vars/kube_control_plane.yml.
  kubelet_custom_flags can no longer be a string, an array is required. (Refactor kubernetes/node templates #10643, @VannTen)
• Action required
  k8s_cluster group is now automatically defined, it can be removed from your inventory if you're not using it for group_vars (Only require minimum structure in inventory, compute the rest #11559, @VannTen)
• Action required
  kubeadm_ignore_preflight_errors is introduced to ignore specific preflight checks from kubeadm. The previous was effectively all, so some errors might surface during upgrade, in which cases, users should add the ones they choose to ignore to that variable. (Stricter kubeadm validation (config and runtime checks) #11710, @VannTen)

Changes by Kind

Container-Managers

• [containerd] Make containerd 1.7.22 default (update containerd 1.7.22 #11554, @foobaar)

API Change

• If you use CRI-O and want to keep runc as your container default runtime when you upgrade cluster, you must set runc_enable: true and crio_default_runtime: "runc".
  Make CRI-O's default runtime configurable
  CRI-O v1.31 default runtime change to crun
  Crun upgrade to 1.17
  Skopeo upgrade to v1.16.1 (Feat: CRI-O v1.31 change default runtime to crun #11601, @tico88612)

Feature

• Add CI for openeuler 24.03
  Add CI Image for openeuler 24.03, 22.03 (Add CI for openEuler 24.03 #11689, @yankay)
• Add a new CRI-O crio_root variable (Add a new crio_root variable in order to store CRI-O data on somethin… #11692, @toliger)
• Add external Oracle cloud infrastructure cloud controller manager (Feat: Add external OCI cloud controller manager #11378, @tico88612)
• Add hashes for kubernetes 1.29.8, 1.29.9, 1.30.5 ([kubernetes] Add hashes for kubernetes 1.29.8, 1.29.9, 1.30.5 #11581, @DirkTheDaring)
• Add optional support for Host Firewall and PolicyAuditMode features in Cilium (Added variables cilium_enable_host_firewall and cilium_policy_audit_mode for configmap/cilium-config #11230, @ledroide)
• Add support Fedora 39/40 (Feat: support Fedora 39/40 #11573, @tico88612)
• Add support to use existing fips with terraform OpenStack (Add support to use exisitng fips with terraform openstack #11558, @anders-elastisys)
• Add the support of network isolation configuration in Multus. (Multus configuration add namespace isolation #11605, @Sispheor)
• Added support for using ntpsec (Add support for ntpsec #11665, @davidumea)
• Adds ingress_nginx_service_annotations variable to allow setting annotations for ingress-nginx controller service (Allow setting annotations on ingress-nginx service #11544, @ThisIsQasim)
• Adds nodelocaldns_additional_configs variable (Adds nodelocaldns_additional_configs to facilitate additional CoreDNS config options #11657, @0x4c6565)
• Allow disabling cilium hubble-ui using cilium_enable_hubble_ui variable (add cilium hubble-ui enable flag #10939, @pedro-peter)
• Configuration can now be supplied to ImagePolicyWebhook and PodNodeSelector admission plugins (Update the list of admission plugins which needs config #11471, @VannTen)
• Default to containerd v1.7.23 (Bump containerd to 1.7.23 #11642, @robertvolkmann)
• Feat(calico): add support for numAllowedLocalASNumbers on bgppeers per node definition (feat(calico): add support for numAllowedLocalASNumbers on bgppeers per node definition #11570, @mirwan)
• Feat: Kubeadm config API support v1beta4 (Feat: kubeadm v1beta4 support #11674, @tico88612)
• Make Kubernetes v1.31.2 default
  Add hashes for Kubernetes 1.30.6 and 1.29.10 (Add hashes for Kubernetes 1.31.2 (new default), 1.30.6 and 1.29.10 #11662, @robertvolkmann)
• Make Kubernetes v1.31.3 default
  Add hashes for Kubernetes 1.30.7 and 1.29.11 (Bump: Kubernetes default version v1.31.3 #11737, @tico88612)
• Make cri-dockerd log level configurable (Make cri-dockerd log level configurable #11646, @mirwan)
• Make kubernetes v1.31.1 default
  Add hashes for Kubernetes 1.31.0 and 1.31.1 ([kubernetes] Support kubernetes 1.31.1 #11533, @philipsabri)
• Partial Support of Cilium v1.16+ - kube-proxy replacement var changes
  Add optional support for configuring BGP Control Plane, IP Load Balancer Pools , Legacy BGP Peer Config v1 and BGP Config v2 features in Cilium (Partial Cilium 1.16+ Support & Add vars for configuring cilium IP load balancer pools and bgp v1 & v2 apis #11620, @logicsys)
• Remove support Fedora 37/38 (Cleanup: Fedora 37/38 CI tests & docs #11600, @tico88612)
• Reset operation: remove /var/log/containers and disable service auto-boot, make sure that multi-user.target.wants is deleted. (Make sure the Reset cleanup is more complete #11501, @leeonfu)
• Support Configuring EncryptionAlgorithm in Kubeadm v1beta4 (Support Configuring EncryptionAlgorithm in Kubeadm v1beta4 #11757, @ErikJiang)
• Update crictl to version v1.31.1 for Kubernetes 1.31
  Update crictl to version v1.30.1 for Kubernetes 1.30 (Add hashes for crictl 1.30.1 and 1.31.1 #11661, @robertvolkmann)
• Update multus to v4.1.0 (Update multus to v4.1.0 and clarify cilium compatibility #11434, @ThisIsQasim)
• Upgrade CoreDNS version to v1.11.3 (Feat: CoreDNS upgrade to v1.11.3 #11653, @tico88612)
• Upgrade OpenStack Cloud Controller Manager to v1.31.1 (Bump: OpenStack Cloud Controller Manager to v1.31.1 #11738, @tico88612)
• Upgrade pause container to 3.10 (Bump: pause container upgrade to 3.10 #11695, @tico88612)
• [cert-manager] upgrade to v1.15.3 ([cert-manager] upgrade to v1.15.3 #11668, @tico88612)
• [cilium] Make cilium 1.15.9 default (update cilium to 1.15.9 #11593, @foobaar)
• [cri-o] Switch binaries to libexecdir
  Update youki version to 0.4.1 to fix ci. ([cri-o] Switch binaries to libexecdir #11584, @yankay)
• [etcd] Default version to 3.5.16 for 1.28, 1.29, 1.30, 1.31 ([etcd] make etcd 3.5.16 default #11572, @janosbabik)
• [need notice] update containerd max_container_log_line_size default value to 16384 (Update containerd containerd_max_container_log_line_size default value to 16384 #11585, @KubeKyrie)
• [nerdctl] Default version to 1.7.7 ([nerdctl] Make nerdctl 1.7.7 default #11575, @janosbabik)
• [runc] Default version to v1.1.14 ([runc] Make runc v1.1.14 default #11574, @janosbabik)

Documentation

• No longer support in-tree cloud provider, please delete or write external to the cloud_provider variable. (Cleanup: remove in-tree cloud provider support #11633, @tico88612)
• Remove inventory_builder scripts and contrib/dind (Remove inventory_builder and re-organize docs #11748, @VannTen)
• Update dns-stack.md reference in docs/ansible/vars.md (Update dns-stack.md reference #11745, @emmanuel-ferdman)

Failing Test

• Fix CI: elastx cleanup security groups (Fix CI: elastx cleanup security groups  #11411, @yankay)
• Update rockylinux9 base image to 9.5 (Update rockylinux9 base image to 9.5 #11744, @yankay)

Bug or Regression

• Action required
  Running kubespray with --limit without cached facts is no longer supported. Improves the scaling for large clusters. (Do not serialize fact gathering for no_proxy #11598, @VannTen)
• Always copy cert generation script to first etcd to pick up fixes on existing clusters (Always copy cert generation scripts to first etcd #11612, @VannTen)
• Fix Cilium agent permission can't read loadbalancerippools and secrets (Update cluster-role for cilium to prevent errors in agent startup #11466, @foobaar)
• Fix collection usage for calico and other configuration depending on .sh and .conf files in Kubespray (Include scripts and conf files in the ansible collection #11707, @VannTen)
• Fix error in boostrap-os when git does not handle symlinks (boostrap-os: use import_tasks instead of symlinks #11508, @VannTen)
• Fix format of kubeadm-config v1beta4 (Remove deprecated key from kubeadmconfig/v1beta4 #11709, @VannTen)
• Fix kube-vip container securityContext (fix kube-vip container securityContext #11647, @KubeKyrie)
• Fix openEuler system packages installation (Simplify package selection + fix openEuler package #11688, @VannTen)
• Fix pretty-printing (in kubectl) of nodelocaldns and coredns configmap when using dns_upstream_forward_extra_opts with an empty value option. (Fix pretty-printing of coredns and nodelocaldns configmaps #11694, @VannTen)
• Fix task naming in bootstrap-os (fix task naming in bootstrap-os #11714, @ErikJiang)
• Fix the check for cached data when using --limit (preinstall: fix checking that excluded host are cached #11693, @VannTen)
• Fix the usage of --limit when using legacy groups (Fix dynamic groups definition when using --limit #11577, @VannTen)
• Fix: cannot stop & remove all cri containers via remove_node.yml (Fix: cannot stop & remove all cri containers via remove_node.yml #11631, @tico88612)
• Fixed: VSphere CSI and CPI drivers and are now retrieved from registry.k8s.io instead of gcr.io, as they have been deleted from the latter. Only a few recent versions are available in the new repository; if you have pinned vsphere_csi_controller, vsphere_csi_driver_image_tag or vsphere_syncer_image_tag to a version older than v3.1.2, please check if that version is available from the new repository. The same goes for external_vsphere_cloud_controller_image_tag which can no longer be latest, and should align with the running version of Kubernetes. It now defaults to v1.31.0. (fix: vsphere image repositories, tags and docs #11564, @luringens)
• HA etcd cluster keeps quorum during upgrades. (etcd: throttle restart for availability #11677, @VannTen)
• Kubeadm images (kube-controller-manager,kube-scheduler,kube-apiserver,kube-proxy) are properly downloaded, including when using the download cache. (Fix the format of the list of images extracted from kubeadm #11741, @VannTen)
• Make sure kubespray-defaults can be executed successfully by executing bootstrap-os first (fix: Swap roles kubespray-defaults & bootstrap-os #11441, @huangkevin404)
• Make upcloud csi_driver use the correct pull secret (Use correct secret name for csi-upcloud pull secret #11597, @VannTen)
• Modifies Helm parameters wait and atomic to be set to false when using kube_network_plugin=cni to prevent deployment issues with kubelet-csr-approver. (Fix kubelet-csr-approver deployment failes if kube_network_plugin=cni(generic) #11704, @M-JavadHeydarpour)
• Remove invalid extraArgs entry and update template file reference (remove nameless extraArgs entry #11703, @agravgaard)
• Update calico-nopde template and remove flexvol-driver initContainer (Fix calico-node template and remove flexvol-driver initContainer #11634, @KubeKyrie)
• Use correct version for community.general collection (Bump version of community.general #11724, @VannTen)

Other (Cleanup or Flake)

• Cleanup older terminology, replace "master" with "control plane" (Cleanup master terminology (start fixing issue #11353) #11394, @bogd)
• Drop support for Kubernetes 1.28.x minimum version now is 1.29.x
  Drop support for CRI-O 1.28.x minimum version now is 1.29.x (Drop support for Kubernetes 1.28.x #11609, @yankay)
• Fix roles/download/tasks/download_file.yml task name typo (Fix task name spelling #11684, @dmncmn)
• Remove pip install . support and rpm spec file (Remove pip and rpm packaging support #11760, @VannTen)
• Replace deprecated unarchive.copy with unarchive.remote_src (replace deprecated unarchive.copy with unarchive.remote_src #11207, @Payback159)
• Update KUBESPRAY_VERSION for v2.26.0 (Update KUBESPRAY_VERSION in CI and Readme for v2.26.0 #11511, @yankay)
• containerd_use_config_path is removed as kubespray now always use containerd config_path configuration. (containerd: always use config_path #11755, @VannTen)

Tasks

Preview Give feedback

1. Add iproute(2) package checking #11816
   approved cncf-cla: yes kind/feature lgtm ok-to-test release-note size/M +7
   
2. Only consider host in 'k8s_cluster' when checking if ip is a cached fact #11817
   approved cncf-cla: yes kind/bug lgtm ok-to-test release-note size/XS +7
   
3. Do not assume systemd-resolved for resolv.conf #11813
   approved cncf-cla: yes kind/bug ok-to-test release-note size/M +6
4. Bump: Kubernetes default version v1.31.4 #11828
   cncf-cla: yes kind/feature ok-to-test release-note size/M +5
5. [calico] Add version 3.29.1 and make it default #11798
   approved cncf-cla: yes kind/feature ok-to-test release-note size/L +6

[VannTen]

Hum I need to fix some of my release notes it seems

[VannTen]

I've updated the release note of my PR, the rendering should be better now.

[yankay]

The Kube 1.32 has been released. https://github.com/kubernetes/kubernetes/releases/tag/v1.32.0
So we need to release kubespray in a few weeks :-)

HI @tico88612
Please update the release notes and create a list of pending issues?
If the pending issue has been fixed, the kubespray will be OK to release :-)

[tico88612]

@yankay I will review the PR & reorganize the release note in this week.

[VannTen]

#11808 Should go in before the release (to unbreak hardening)

[VannTen]

(EDIT: -> put the fixes in the tasklist (that feature is pretty good 🤔 )

[yankay]

It needs some bumps

• helm from v3.15.4 to v3.16.4
• containerd from v1.7.23 to 1.7.24
• runc from v1.1.14 to v1.2.3 ?
• kubernetes from v1.31.3 to v1.31.4