Add optional highAvailability feature for registry cache #298

dergeberl · 2024-12-04T13:57:05Z

How to categorize this PR?

/area high-availability
/kind enhancement

What this PR does / why we need it:

I reopend this PR due to some special A in the branch name of the other PR (#247)
This PR adds the optional highAvailability setting for the proxy cache. As mentioned in #246 in some cases the registry cache becomes more critical (like the proxy use case). In such cases it makes sense to scale the registry cache to more replicas. This PR uses the High Availability of Deployed Components feature to also set a working topologySpreadConstraints.

Which issue(s) this PR fixes:
N/A

Special notes for your reviewer:

Release note:

Add optional setting for a registry cache to scale the `StatefulSet` to 2 replicas with the [High Availability of Deployed Components](https://gardener.cloud/docs/gardener/high-availability/#system-components) feature.

gardener-ci-robot · 2024-12-25T12:23:12Z

The Gardener project currently lacks enough active contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:

After 15d of inactivity, lifecycle/stale is applied
After 15d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 7d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Mark this PR as rotten with /lifecycle rotten
Close this PR with /close

/lifecycle stale

dergeberl · 2024-12-27T12:30:33Z

/remove-lifecycle stale

gardener-ci-robot · 2025-01-11T13:35:43Z

The Gardener project currently lacks enough active contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:

After 15d of inactivity, lifecycle/stale is applied
After 15d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 7d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Mark this PR as rotten with /lifecycle rotten
Close this PR with /close

/lifecycle stale

dimitar-kostadinov · 2025-01-13T06:50:49Z

/remove-lifecycle stale

ialidzhikov · 2025-01-13T14:36:01Z

go: github.com/gardener/[email protected]: Get "http://athens-proxy.athens.svc.cluster.local/github.com/gardener/gardener/@v/v1.110.0.info": dial tcp 100.111.158.151:80: connect: connection refused
go: invalid module version syntax "github.com/gardener/gardener@"
go: github.com/gardener/[email protected]: Get "http://athens-proxy.athens.svc.cluster.local/github.com/gardener/gardener/@v/v1.110.0.info": dial tcp 100.111.158.151:80: connect: connection refused
bash: /hack/get-build-ld-flags.sh: No such file or directory
Makefile:34: /hack/tools.mk: No such file or directory
make: *** No rule to make target '/hack/tools.mk'.  Stop.

/test pull-gardener-extension-registry-cache-unit

ialidzhikov · 2025-01-13T13:59:22Z

docs/usage/registry-cache/configuration.md

-The registry cache runs with a single replica. This fact may lead to concerns for the high availability such as "What happens when the registry cache is down? Does containerd fail to pull the image?". As outlined in the [How does it work? section](#how-does-it-work), containerd is configured to fall back to the upstream registry if it fails to pull the image from the registry cache. Hence, when the registry cache is unavailable, the containerd's image pull operations are not affected because containerd falls back to image pull from the upstream registry.
+Per default the registry cache runs with a single replica. This fact may lead to concerns for the high availability such as "What happens when the registry cache is down? Does containerd fail to pull the image?". As outlined in the [How does it work? section](#how-does-it-work), containerd is configured to fall back to the upstream registry if it fails to pull the image from the registry cache. Hence, when the registry cache is unavailable, the containerd's image pull operations are not affected because containerd falls back to image pull from the upstream registry.
+
+In special cases where this is not enough (for example when using the registry cache with a proxy) it is possible to set `providerConfig.caches[].highAvailability` to `true`, this will add the label `high-availability-config.resources.gardener.cloud/type=server` and scale the statefulset to 2 replicas. The `topologySpreadConstraints` is added according to the cluster configuration. See also [High Availability of Deployed Components](https://gardener.cloud/docs/gardener/high-availability/#system-components).


Can you elaborate more on the use case that requires running the registry cache in HA mode? I am not able to understand how a registry cache running against an upstream behind a proxy requires the registry cache to run in HA mode.

In my example I mean if the upstream registry is only accessible via a proxy no image can be pulled i the cache is not reachable.

Is my understanding correct that:

When using a registry behind a proxy it is not easy/possible to set http proxy settings to containerd. And with this feature, you would like to run the registry cache with 2 replicas and with this you would reduce the probability of having the registry-cache down and prevent containerd from falling back to the upstream.

?

If that is the case, I am not sure if it is a good solution to the problem.

How the initial Shoot creation with enabled registry-cache extension works in the case containerd is not able to fall back to the upstream? As described in https://github.com/gardener/gardener-extension-registry-cache/blob/main/docs/usage/registry-cache/configuration.md#limitations, we cannot cache all images from Shoot system components (because of the design decision that the registry-cache runs in the cluster and requires the Pod and Service network to be set up). If you create a Shoot with enabled registry-cache extension for such an upstream, and if indeed containerd cannot fall back to the upstream due to missing http proxy config, then Shoot creation wouldn't work.

The registry caches also rely on volumes. Having attach/detach issues with the volume would again lead to not having a running registry cache Pod and would again lead to image pull failures from containerd.

Let me know if my assumptions are correct.

If yes, IMO it would make much more sense to set http proxy settings to containerd instead of enabling HA for the registry cache StatefulSet.

docs/usage/registry-cache/configuration.md

pkg/apis/registry/types.go

pkg/component/registrycaches/registry_caches.go

pkg/apis/registry/types.go

pkg/apis/registry/v1alpha3/types.go

docs/usage/registry-cache/configuration.md

pkg/component/registrycaches/registry_caches.go

ialidzhikov · 2025-01-13T14:42:53Z

I reopend this PR due to some special A in the branch name of the other PR (#247)

This seems to be caused by #313. 🙈
We are sorry for the trouble!

ialidzhikov · 2025-01-13T14:43:55Z

e2e tests job also failed due to network issues.

/test pull-gardener-extension-registry-cache-e2e-kind

ialidzhikov · 2025-01-14T10:24:18Z

The e2e test failure is occurrence of #283.

ialidzhikov

We could also add e2e test to cover this scenario. @dimitar-kostadinov raised the point that helper funcs that perform the checks have to be adapted to consider now the 2 replicas of the StatefulSet.

gardener-prow · 2025-01-29T13:58:16Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from dimitar-kostadinov. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dergeberl · 2025-01-29T14:12:32Z

@ialidzhikov Thanks for your feedback.

I added your feedback from the PR review.

Except the E2E test, so you mean an E2E test with highAvailability enabled and check if 2 pods are running?

gardener-prow · 2025-02-05T14:21:40Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dimitar-kostadinov

One unused func to be cleanup, otherwise
/lgtm

@dergeberl can you rebase the PR?

dimitar-kostadinov · 2025-02-21T10:05:56Z

pkg/component/registrycaches/registry_caches_test.go

+			_ = func(name, upstream, remoteURL string) *corev1.Service {
+				return &corev1.Service{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      name,
+						Namespace: "kube-system",
+						Labels: map[string]string{
+							"app":           name,
+							"upstream-host": upstream,
+						},
+						Annotations: map[string]string{
+							"upstream":   upstream,
+							"remote-url": remoteURL,
+						},
+					},
+					Spec: corev1.ServiceSpec{
+						Selector: map[string]string{
+							"app":           name,
+							"upstream-host": upstream,
+						},
+						Ports: []corev1.ServicePort{{
+							Name:       "registry-cache",
+							Port:       5000,
+							Protocol:   corev1.ProtocolTCP,
+							TargetPort: intstr.FromString("registry-cache"),
+						}},
+						Type: corev1.ServiceTypeClusterIP,
+					},
+				}
+			}


These lines can be removed as registry cache services are being moved to a separate component and this function is now here.

gardener-prow · 2025-02-21T10:17:45Z

LGTM label has been added.

Git tree hash: d8822ea739cd78018294c09e13efde39e485236d

ialidzhikov · 2025-02-24T13:25:37Z

docs/usage/registry-cache/configuration.md

-The registry cache runs with a single replica. This fact may lead to concerns for the high availability such as "What happens when the registry cache is down? Does containerd fail to pull the image?". As outlined in the [How does it work? section](#how-does-it-work), containerd is configured to fall back to the upstream registry if it fails to pull the image from the registry cache. Hence, when the registry cache is unavailable, the containerd's image pull operations are not affected because containerd falls back to image pull from the upstream registry.
+Per default the registry cache runs with a single replica. This fact may lead to concerns for the high availability such as "What happens when the registry cache is down? Does containerd fail to pull the image?". As outlined in the [How does it work? section](#how-does-it-work), containerd is configured to fall back to the upstream registry if it fails to pull the image from the registry cache. Hence, when the registry cache is unavailable, the containerd's image pull operations are not affected because containerd falls back to image pull from the upstream registry.
+
+In special cases where this is not enough (for example when using an upstream which is only accessible with a proxy) it is possible to set `providerConfig.caches[].highAvailability` to `true`, this will add the label `high-availability-config.resources.gardener.cloud/type=server` and scale the statefulset to 2 replicas. The `topologySpreadConstraints` is added according to the cluster configuration. See also [High Availability of Deployed Components](https://gardener.cloud/docs/gardener/high-availability/#system-components). Each registry cache replica uses an own volume, so each registry cache needs to pull the image from upstream.


Suggested change

In special cases where this is not enough (for example when using an upstream which is only accessible with a proxy) it is possible to set `providerConfig.caches[].highAvailability` to `true`, this will add the label `high-availability-config.resources.gardener.cloud/type=server` and scale the statefulset to 2 replicas. The `topologySpreadConstraints` is added according to the cluster configuration. See also [High Availability of Deployed Components](https://gardener.cloud/docs/gardener/high-availability/#system-components). Each registry cache replica uses an own volume, so each registry cache needs to pull the image from upstream.

In special cases where this is not enough (for example when using an upstream which is only accessible with a proxy) it is possible to set `providerConfig.caches[].highAvailability.enabled` to `true`. This will add the label `high-availability-config.resources.gardener.cloud/type=server` to the StatefulSet and it will be scaled to 2 replicas. Appropriate [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) will be added to the registry cache Pods according to the Shoot cluster configuration. See also [High Availability of Deployed Components](https://github.com/gardener/gardener/blob/master/docs/development/high-availability-of-components.md#system-components). Each registry cache replica uses an own volume, so each registry cache needs to pull the image from the upstream.

Few nits:

The field name is now providerConfig.caches[].highAvailability.enabled

We link gardener/gardener docs via github, not website (gardener.cloud) - see Fix links to gardener/gardener docs #158

Other working suggestions

ialidzhikov · 2025-02-24T13:45:14Z

docs/usage/registry-cache/configuration.md

-The registry cache runs with a single replica. This fact may lead to concerns for the high availability such as "What happens when the registry cache is down? Does containerd fail to pull the image?". As outlined in the [How does it work? section](#how-does-it-work), containerd is configured to fall back to the upstream registry if it fails to pull the image from the registry cache. Hence, when the registry cache is unavailable, the containerd's image pull operations are not affected because containerd falls back to image pull from the upstream registry.
+Per default the registry cache runs with a single replica. This fact may lead to concerns for the high availability such as "What happens when the registry cache is down? Does containerd fail to pull the image?". As outlined in the [How does it work? section](#how-does-it-work), containerd is configured to fall back to the upstream registry if it fails to pull the image from the registry cache. Hence, when the registry cache is unavailable, the containerd's image pull operations are not affected because containerd falls back to image pull from the upstream registry.
+
+In special cases where this is not enough (for example when using the registry cache with a proxy) it is possible to set `providerConfig.caches[].highAvailability` to `true`, this will add the label `high-availability-config.resources.gardener.cloud/type=server` and scale the statefulset to 2 replicas. The `topologySpreadConstraints` is added according to the cluster configuration. See also [High Availability of Deployed Components](https://gardener.cloud/docs/gardener/high-availability/#system-components).


Is my understanding correct that:

When using a registry behind a proxy it is not easy/possible to set http proxy settings to containerd. And with this feature, you would like to run the registry cache with 2 replicas and with this you would reduce the probability of having the registry-cache down and prevent containerd from falling back to the upstream.

?

If that is the case, I am not sure if it is a good solution to the problem.

How the initial Shoot creation with enabled registry-cache extension works in the case containerd is not able to fall back to the upstream? As described in https://github.com/gardener/gardener-extension-registry-cache/blob/main/docs/usage/registry-cache/configuration.md#limitations, we cannot cache all images from Shoot system components (because of the design decision that the registry-cache runs in the cluster and requires the Pod and Service network to be set up). If you create a Shoot with enabled registry-cache extension for such an upstream, and if indeed containerd cannot fall back to the upstream due to missing http proxy config, then Shoot creation wouldn't work.

The registry caches also rely on volumes. Having attach/detach issues with the volume would again lead to not having a running registry cache Pod and would again lead to image pull failures from containerd.

Let me know if my assumptions are correct.

If yes, IMO it would make much more sense to set http proxy settings to containerd instead of enabling HA for the registry cache StatefulSet.

gardener-prow bot added area/high-availability High availability related kind/enhancement Enhancement, improvement, extension labels Dec 4, 2024

gardener-prow bot requested review from dimitar-kostadinov and ialidzhikov December 4, 2024 13:57

gardener-prow bot added cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 4, 2024

dergeberl mentioned this pull request Dec 4, 2024

Add optional highAvailability feature for registry cache #247

Closed

gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 10, 2024

gardener-prow bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 25, 2024

gardener-prow bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 27, 2024

dergeberl force-pushed the highAvailabilityProxyCache branch from 3fe41e1 to 2c2264c Compare December 27, 2024 13:02

gardener-prow bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 27, 2024

gardener-prow bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 11, 2025

gardener-prow bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 13, 2025

ialidzhikov assigned ialidzhikov and dimitar-kostadinov Jan 13, 2025

ialidzhikov reviewed Jan 13, 2025

View reviewed changes

ialidzhikov reviewed Jan 15, 2025

View reviewed changes

gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 25, 2025

dergeberl and others added 2 commits January 29, 2025 14:31

Add optional highAvailability feature for registry cache

445271e

Add PR feedback to docs

82505d2

dergeberl force-pushed the highAvailabilityProxyCache branch from 2c2264c to 5c1c22a Compare January 29, 2025 13:58

gardener-prow bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 29, 2025

dergeberl added 2 commits January 29, 2025 15:01

Use different API structure highAvailability.enabled

386f5c7

Apply different implementation for PR review

853b68b

dergeberl force-pushed the highAvailabilityProxyCache branch from 5c1c22a to 853b68b Compare January 29, 2025 14:05

gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 5, 2025

ialidzhikov self-requested a review February 17, 2025 09:05

dimitar-kostadinov reviewed Feb 21, 2025

View reviewed changes

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Feb 21, 2025

ialidzhikov reviewed Feb 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add optional highAvailability feature for registry cache #298

Add optional highAvailability feature for registry cache #298

dergeberl commented Dec 4, 2024

gardener-ci-robot commented Dec 25, 2024

dergeberl commented Dec 27, 2024

gardener-ci-robot commented Jan 11, 2025

dimitar-kostadinov commented Jan 13, 2025

ialidzhikov commented Jan 13, 2025

ialidzhikov Jan 13, 2025

dergeberl Jan 29, 2025

ialidzhikov Feb 24, 2025

ialidzhikov commented Jan 13, 2025

ialidzhikov commented Jan 13, 2025

ialidzhikov commented Jan 14, 2025

ialidzhikov left a comment

gardener-prow bot commented Jan 29, 2025

dergeberl commented Jan 29, 2025

gardener-prow bot commented Feb 5, 2025

dimitar-kostadinov left a comment

dimitar-kostadinov Feb 21, 2025

gardener-prow bot commented Feb 21, 2025

ialidzhikov Feb 24, 2025

ialidzhikov Feb 24, 2025

Add optional highAvailability feature for registry cache #298

Are you sure you want to change the base?

Add optional highAvailability feature for registry cache #298

Conversation

dergeberl commented Dec 4, 2024

gardener-ci-robot commented Dec 25, 2024

dergeberl commented Dec 27, 2024

gardener-ci-robot commented Jan 11, 2025

dimitar-kostadinov commented Jan 13, 2025

ialidzhikov commented Jan 13, 2025

ialidzhikov Jan 13, 2025

Choose a reason for hiding this comment

dergeberl Jan 29, 2025

Choose a reason for hiding this comment

ialidzhikov Feb 24, 2025

Choose a reason for hiding this comment

ialidzhikov commented Jan 13, 2025

ialidzhikov commented Jan 13, 2025

ialidzhikov commented Jan 14, 2025

ialidzhikov left a comment

Choose a reason for hiding this comment

gardener-prow bot commented Jan 29, 2025

dergeberl commented Jan 29, 2025

gardener-prow bot commented Feb 5, 2025

dimitar-kostadinov left a comment

Choose a reason for hiding this comment

dimitar-kostadinov Feb 21, 2025

Choose a reason for hiding this comment

gardener-prow bot commented Feb 21, 2025

ialidzhikov Feb 24, 2025

Choose a reason for hiding this comment

ialidzhikov Feb 24, 2025

Choose a reason for hiding this comment