[FEATURE]: Use Internal k8sUtil Image #6877

msharma38 · 2025-02-24T13:13:21Z

Feature Request

We want to reference k8sUtil image rom out internal registry not from docker hub.

What should the user be able to do?

Reference k8sUtil image rom out internal registry i.e GAR or artifactory

Why do they want to do this? What problem does it solve?

We are hitting into below issue where garden is trying to pull the image from open source docker hub and getting below issue.

failed to resolve resolvedRef at path /patchStrategicMerge/spec/template/spec/containers/0/image: failed to fetch image descriptor: gardendev/k8s-util:0.5.7@sha256:522da245a5e6ae7c711aa94f84fc83f82a8fdffbf6d8bc48f4d80fee0e0e631b, error: failed to fetch image descriptor: gardendev/k8s-util:0.5.7@sha256:522da245a5e6ae7c711aa94f84fc83f82a8fdffbf6d8bc48f4d80fee0e0e631b, error: failed to fetch image reference: gardendev/k8s-util:0.5.7@sha256:522da245a5e6ae7c711aa94f84fc83f82a8fdffbf6d8bc48f4d80fee0e0e631b, error: GET https://index.docker.io/v2/gardendev/k8s-util/manifests/sha256:522da245a5e6ae7c711aa94f84fc83f82a8fdffbf6d8bc48f4d80fee0e0e631b: TOOMANYREQUESTS: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit]

Suggested Implementation(s)

give an option for users to use pass k8sUtil image path.

How important is this feature for you/your team?

🥀 Crucial, Garden is unusable for us without it

vvagaytsev · 2025-02-25T12:33:52Z

@msharma38 thanks for reporting this!

Ability to use a custom registry mirror for utility images was implemented in #6552.

Please check the guide at https://docs.garden.io/kubernetes-plugins/guides/registry-mirror
See also https://docs.garden.io/misc/faq#how-do-i-avoid-being-rate-limited-by-docker-hub

We hope that helps. Please let use know if you have any troubles with that approach.

vvagaytsev · 2025-02-25T12:45:27Z

@msharma38 I'm going to close this issue. Feel free to reopen it if the issue persists.

msharma38 · 2025-02-26T03:20:51Z

HI @vvagaytsev after we have imported the image inside our GAR and attesting it the digest is different for what it is present in docker hub
image digest in docker hub :- sha256:0f41babfcfa216fe23561195c5c096aff6d9c374cd8850da12b6e00bc944491b
In our GAR :- sha256:9b37c4f031157750c02fec36152a1b39e3d04dbbd281f0bfe54466360ed4f01a

as we are referring the digest from docker hub in image which is not present in my internal repo the build is failing with below error

Kubernetes Message: admission webhook "mutate.kyverno.svc-fail" denied the request: mutation policy resolve-image-to-digest error: failed to apply policy resolve-image-to-digest rules [autogen-resolve-to-digest: failed to mutate elements: variable substitution failed: failed to resolve resolvedRef at path /patchStrategicMerge/spec/template/spec/containers/0/image: failed to fetch image descriptor: us-docker.pkg.dev/wf-gcp-us-plat-gar-prod/external-staging/gardendev/k8s-util:0.6.4@sha256:0f41babfcfa216fe23561195c5c096aff6d9c374cd8850da12b6e00bc944491b, error: failed to fetch image descriptor: us-docker.pkg.dev/wf-gcp-us-plat-gar-prod/external-staging/gardendev/k8s-util:0.6.4@sha256:0f41babfcfa216fe23561195c5c096aff6d9c374cd8850da12b6e00bc944491b, error: failed to fetch image reference: us-docker.pkg.dev/wf-gcp-us-plat-gar-prod/external-staging/gardendev/k8s-util:0.6.4@sha256:0f41babfcfa216fe23561195c5c096aff6d9c374cd8850da12b6e00bc944491b, error: GET https://us-docker.pkg.dev/v2/wf-gcp-us-plat-gar-prod/external-staging/gardendev/k8s-util/manifests/sha256:0f41babfcfa216fe23561195c5c096aff6d9c374cd8850da12b6e00bc944491b: MANIFEST_UNKNOWN: Requested entity was not found.]

stefreak · 2025-02-26T18:30:16Z

@msharma38 did you rebuild the image to upload it to GAR? If you copy the image using a tool like regctl (regctl copy) then the image digest should stay the same

stefreak · 2025-02-26T18:32:17Z

GAR also allows you to set up a pull-through cache (they call that a "remote repo"), is that maybe what you need?

msharma38 · 2025-02-27T05:50:57Z

@stefreak we are running gcloud beta container binauthz attestations sign-and-create command to attest the images and push it to GAR.

msharma38 · 2025-03-12T09:52:18Z

HI @stefreak i am trying to update to garden 0.13.54 and getting below error
Error:-

[k8s-buildkite-agent stage-1 9/9] RUN cd /tmp/garden_init && git init && garden util fetch-tools:

| 0.349 hint: Using 'master' as the name for the initial branch. This default branch name
| 0.349 hint: is subject to change. To configure the initial branch name to use in all
| 0.349 hint: of your new repositories, which will suppress this warning, call:
| 0.349 hint:
| 0.349 hint: git config --global init.defaultBranch
| 0.349 hint:
| 0.349 hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
| 0.349 hint: 'development'. The just-created branch can be renamed via this command:
| 0.349 hint:
| 0.349 hint: git branch -m
| 0.350 Initialized empty Git repository in /tmp/garden_init/.git/
| 1.576 Error relocating /home/buildkite/.local/share/garden/1741680672-wSzfzLh.r/bin/node: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE15_M_replace_coldEPcmPKcmm: symbol not found
| ------
| Traceback (most recent call last):
| File "/app/run_all_from_dockercompose.py", line 460, in
| cli() # pylint: disable=no-value-for-parameter
| File "/pyenv/versions/3.9.15/lib/python3.9/site-packages/click/core.py", line 1157, in call
| return self.main(*args, **kwargs)
| File "/pyenv/versions/3.9.15/lib/python3.9/site-packages/click/core.py", line 1078, in main
| rv = self.invoke(ctx)
| File "/pyenv/versions/3.9.15/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
| return ctx.invoke(self.callback, **ctx.params)
| File "/pyenv/versions/3.9.15/lib/python3.9/site-packages/click/core.py", line 783, in invoke
| return __callback(*args, **kwargs)
| File "/app/run_all_from_dockercompose.py", line 362, in cli
| build(
| File "/app/wf_docker/build.py", line 282, in build
| client.compose.build(services=[service_name], build_args=custom_args) # type: ignore
| File "/pyenv/versions/3.9.15/lib/python3.9/site-packages/python_on_whales/components/compose/cli_wrapper.py", line 62, in build
| run(full_cmd, capture_stdout=False)
| File "/pyenv/versions/3.9.15/lib/python3.9/site-packages/python_on_whales/utils.py", line 194, in run
| raise DockerException(
| python_on_whales.exceptions.DockerException: The docker command executed was /usr/bin/docker compose --file docker-composea7a31e2e-e431-485d-b1d4-89b335c7b429.yaml build --build-arg BK_HTTP_PROXY=http://bkegress.service.intraiad1.consul.csnzoo.com:3128 --build-arg BK_HTTPS_PROXY=http://bkegress.service.intraiad1.consul.csnzoo.com:3128 k8s-buildkite-agent.
| It returned with code 17
| The content of stdout can be found above the stacktrace (it wasn't captured).
| The content of stderr is 'failed to solve: executor failed running [/bin/sh -c cd /tmp/garden_init && git init && garden util fetch-tools]: exit code: 127

We are using alpine:3.15.0 as base image this was working with garden 0.13.35 version

stefreak · 2025-03-12T12:43:08Z

@msharma38 I'm not quite sure what I'm looking at here, this looks like an error output from Docker compose. Can you share the Dockerfile with us that fails to build here?

msharma38 · 2025-03-12T13:33:03Z

@stefreak below is the docker file which is working for 0.13.35 version of garden

ARG BK_HTTP_PROXY
ARG BK_HTTPS_PROXY


FROM alpine:3.15.0 as builder
ARG GH_VERSION="2.30.0"
RUN apk update \
    && apk add --no-cache wget rsync \
    && wget https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_amd64.tar.gz \
    && tar -zxvf gh_${GH_VERSION}_linux_amd64.tar.gz \
    && chmod +x gh_${GH_VERSION}_linux_amd64/bin/gh \
    && rsync -az --remove-source-files gh_${GH_VERSION}_linux_amd64/bin/ /usr/bin

FROM external-staging/buildkite/agent:3.49.0
RUN apk add nodejs npm
RUN echo "Node Version:" && node -v
RUN apk add --no-cache coreutils
ARG http_proxy=${BK_HTTP_PROXY}
ARG https_proxy=${BK_HTTPS_PROXY}
ARG argocd_ver="v2.7.4"
ARG kubectl_ver="v1.26.12"
ARG helm3_ver="v3.13.3"
ARG helm2_ver="v2.17.0"
ARG helmfile_ver="0.160.0"
ARG kustomize_ver="v5.3.0"
ENV wf_version="3.49.0-wf6"
ENV wf_description="Buildkite Agent for Kubernetes"
ENV HELM_DRIVER="configmap"
ENV BUILDKITE_AGENT_CONFIG="/home/buildkite/buildkite-agent.cfg"

LABEL \
  com.wayfair.app="wayfair/k8s-buildkite-agent" \
  com.wayfair.description=${wf_description} \
  com.wayfair.maintainer="Kubernetes <[email protected]>" \
  com.wayfair.vendor="Wayfair LLC." \
  com.wayfair.version=${wf_version}

COPY --from=builder /usr/bin/gh /usr/bin/
COPY --from=hairyhenderson/gomplate:v3.10.0-alpine /bin/gomplate /bin/gomplate
COPY --from=wayfair/centos82-base:1.0.11 /etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt

RUN echo "Starting build" \
  && adduser -h /home/buildkite -s /sbin/nologin -u 1001 -D -S -g buildkite buildkite \
  && rsync -a /buildkite/* /home/buildkite/ \
  && cat /buildkite/buildkite-agent.cfg | sed -e 's#="/buildkite#="/home/buildkite#' > /home/buildkite/buildkite-agent.cfg \
  && curl -L https://dl.k8s.io/release/${kubectl_ver}/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl \
  && curl -L https://get.helm.sh/helm-${helm3_ver}-linux-amd64.tar.gz | tar xvz -C /usr/local/bin/ linux-amd64/helm --strip-components 1 \
  && mv /usr/local/bin/helm /usr/local/bin/helm3 \
  && curl -L https://get.helm.sh/helm-${helm2_ver}-linux-amd64.tar.gz | tar xvz -C /usr/local/bin/ linux-amd64/helm --strip-components 1 \
  && curl -L https://github.com/helmfile/helmfile/releases/download/v${helmfile_ver}/helmfile_${helmfile_ver}_linux_amd64.tar.gz -o helmfile.tar.gz && tar xvzf helmfile.tar.gz helmfile -C /usr/local/bin/ \
  && chmod 0755 /usr/local/bin/kubectl /usr/local/bin/helm3 /usr/local/bin/helm /usr/local/bin/helmfile \
  && curl -L https://github.com/mikefarah/yq/releases/download/v4.22.1/yq_linux_amd64 -o /usr/local/bin/yq \
  && chmod 0755 /usr/local/bin/yq \
  && curl -L https://releases.jfrog.io/artifactory/jfrog-cli/v2-jf/2.13.0/jfrog-cli-linux-amd64/jf -o /usr/local/bin/jf \
  && chmod 0755 /usr/local/bin/jf \
  && su -s /bin/bash buildkite -c '/usr/local/bin/helm init --client-only' \
  && su -s /bin/bash buildkite -c '/usr/local/bin/helm repo add helm-virtual https://artifactorydocker.service.csnzoo.com/artifactory/helm-virtual' \
  && curl -L https://github.com/databus23/helm-diff/releases/download/v2.11.0%2B5/helm-diff-linux.tgz  | tar xzv -C /home/buildkite/.helm/plugins \
  && mkdir -p /home/buildkite/.local/share/helm/plugins \
  && curl -L https://artifactorybase.service.csnzoo.com/artifactory/helm-plugins/diff/diff-v3.4.2.tgz | tar xvz -C /home/buildkite/.local/share/helm/plugins/ \
  && curl -L https://artifactorybase.service.csnzoo.com/artifactory/helm-plugins/2to3/2to3-v0.10.0.tgz | tar xvz -C /home/buildkite/.local/share/helm/plugins/ \
  && curl -L https://github.com/argoproj/argo-cd/releases/download/${argocd_ver}/argocd-linux-amd64 -o /usr/local/bin/argocd && chmod +x /usr/local/bin/argocd \
  && chown -R buildkite: /home/buildkite \
  && curl -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${kustomize_ver}/kustomize_${kustomize_ver}_linux_amd64.tar.gz -o kustomize.tar.gz && tar xvzf kustomize.tar.gz kustomize -C /usr/local/bin/ \
  && chmod 0755 /usr/local/bin/kustomize \
  # Install gcloud SDK
  && apk add --no-cache curl bash python3 \
  && curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-426.0.0-linux-x86_64.tar.gz \
  && tar zxvf google-cloud-sdk-426.0.0-linux-x86_64.tar.gz -C /usr/local \
  && chmod -R 755 /usr/local/google-cloud-sdk \
  && /usr/local/google-cloud-sdk/install.sh --quiet \
  && rm google-cloud-sdk-426.0.0-linux-x86_64.tar.gz \
  && ln -s /usr/local/google-cloud-sdk/bin/gcloud /usr/bin/gcloud \
  && ln -s /usr/local/google-cloud-sdk/bin/gsutil /usr/bin/gsutil \
  && ln -s /usr/local/google-cloud-sdk/bin/bq /usr/bin/bq \
  # Install beta components
  && /usr/local/google-cloud-sdk/bin/gcloud components install beta --quiet \
  #Install gcloud-auth-plugin
  && /usr/local/google-cloud-sdk/bin/gcloud components install gke-gcloud-auth-plugin --quiet \
  && echo "export USE_GKE_GCLOUD_AUTH_PLUGIN=True" >> /etc/profile \
  && echo "Done"

#Install garden
RUN curl -LO https://artifactorybase.service.csnzoo.com/artifactory/garden_vrtsnc/garden/garden/wf/prod/v0.13.54/dist/garden-0.13.54-alpine-amd64.tar.gz && \
    mkdir -p /tmp/garden-install && \
    tar -xzf garden-0.13.54-alpine-amd64.tar.gz -C /tmp/garden-install && \
    rm garden-0.13.54-alpine-amd64.tar.gz && \
    mkdir -p /home/buildkite/.garden/bin && \
    cp -r /tmp/garden-install/alpine-amd64/* /home/buildkite/.garden/bin && \
    ln -s /home/buildkite/.garden/bin/garden /usr/local/bin/garden \
    && chown -R buildkite: /home/buildkite \
    && echo "Done"

# Installl go
RUN curl -O https://dl.google.com/go/go1.21.9.linux-amd64.tar.gz \
    && tar zxvf go1.21.9.linux-amd64.tar.gz -C /usr/local \ 
    && mkdir -p /usr/local/go/src/configmap \
    && chmod o+rwx /usr/local/go/src/configmap \
    && echo "Done"

RUN apk update && apk add libstdc++6

USER 1001

RUN mkdir -p /tmp/garden_init
COPY files/garden.base.yaml /tmp/garden_init/garden.yaml

RUN cd /tmp/garden_init \
    && git init && garden util fetch-tools

msharma38 · 2025-03-12T18:10:45Z

@stefreak did you checked the above file as we are blocked completely and one more question is there any way we can specify the k8's util image in garden

stefreak · 2025-03-13T07:33:46Z

@msharma38 I'm sorry this is blocking your team. You can specify the util image registry in Garden (Docs are here).

What you're doing isn't supported at the moment in the Open-Source Garden. The latest versions of Garden pin the SHA256 of each Docker image it ships with to prevent inadvertently using wrong or modified images.

I didn't fully comprehend your needs yet, but it seems like a feature request to me, as you seem to have special requirements on the provenance of the Garden Docker images, and the requirement that the attestations are signed.

We do accept contributions, so you you're happy to implement what you need and contribute.

We also offer Enterprise support plans (Contact us at https://garden.io/contact) where we also do solution engineering. Here on GitHub on the other hand we're providing best-effort support to the community and small-scale users.

msharma38 · 2025-03-13T07:37:31Z

@stefreak did you get the chance to look into the docker file for the error

stefreak · 2025-03-13T07:39:09Z

Yes, as I've said the latest versions of Garden pin the SHA256 of each Docker image it ships with to prevent inadvertently using wrong or modified images. This is what is causing the error.

stefreak · 2025-03-13T07:42:42Z

The easiest solution that I can see is using the artifact registry "remote repo" feature, as it will preserve the SHA256 of the images. https://cloud.google.com/artifact-registry/docs/repositories/remote-repo

msharma38 · 2025-03-13T07:47:03Z

@stefreak we are updating garden version from 0.13.35 to 0.13.54 but while doing that we are getting the below error

Error :- | 1.576 Error relocating /home/buildkite/.local/share/garden/1741680672-wSzfzLh.r/bin/node: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE15_M_replace_coldEPcmPKcmm: symbol not found

Docker file :-

ARG BK_HTTP_PROXY
ARG BK_HTTPS_PROXY

FROM alpine:3.15.0 as builder
ARG GH_VERSION="2.30.0"
RUN apk update
&& apk add --no-cache wget rsync
&& wget https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_amd64.tar.gz
&& tar -zxvf gh_${GH_VERSION}linux_amd64.tar.gz
&& chmod +x gh${GH_VERSION}linux_amd64/bin/gh
&& rsync -az --remove-source-files gh${GH_VERSION}_linux_amd64/bin/ /usr/bin

FROM external-staging/buildkite/agent:3.49.0
RUN apk add nodejs npm
RUN echo "Node Version:" && node -v
RUN apk add --no-cache coreutils
ARG http_proxy=${BK_HTTP_PROXY}
ARG https_proxy=${BK_HTTPS_PROXY}
ARG argocd_ver="v2.7.4"
ARG kubectl_ver="v1.26.12"
ARG helm3_ver="v3.13.3"
ARG helm2_ver="v2.17.0"
ARG helmfile_ver="0.160.0"
ARG kustomize_ver="v5.3.0"
ENV wf_version="3.49.0-wf6"
ENV wf_description="Buildkite Agent for Kubernetes"
ENV HELM_DRIVER="configmap"
ENV BUILDKITE_AGENT_CONFIG="/home/buildkite/buildkite-agent.cfg"

LABEL
com.wayfair.app="wayfair/k8s-buildkite-agent"
com.wayfair.description=${wf_description}
com.wayfair.maintainer="Kubernetes [email protected]"
com.wayfair.vendor="Wayfair LLC."
com.wayfair.version=${wf_version}

COPY --from=builder /usr/bin/gh /usr/bin/
COPY --from=hairyhenderson/gomplate:v3.10.0-alpine /bin/gomplate /bin/gomplate
COPY --from=wayfair/centos82-base:1.0.11 /etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt

RUN echo "Starting build"
&& adduser -h /home/buildkite -s /sbin/nologin -u 1001 -D -S -g buildkite buildkite
&& rsync -a /buildkite/* /home/buildkite/
&& cat /buildkite/buildkite-agent.cfg | sed -e 's#="/buildkite#="/home/buildkite#' > /home/buildkite/buildkite-agent.cfg
&& curl -L https://dl.k8s.io/release/${kubectl_ver}/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl
&& curl -L https://get.helm.sh/helm-${helm3_ver}-linux-amd64.tar.gz | tar xvz -C /usr/local/bin/ linux-amd64/helm --strip-components 1
&& mv /usr/local/bin/helm /usr/local/bin/helm3
&& curl -L https://get.helm.sh/helm-${helm2_ver}-linux-amd64.tar.gz | tar xvz -C /usr/local/bin/ linux-amd64/helm --strip-components 1
&& curl -L https://github.com/helmfile/helmfile/releases/download/v${helmfile_ver}/helmfile_${helmfile_ver}_linux_amd64.tar.gz -o helmfile.tar.gz && tar xvzf helmfile.tar.gz helmfile -C /usr/local/bin/
&& chmod 0755 /usr/local/bin/kubectl /usr/local/bin/helm3 /usr/local/bin/helm /usr/local/bin/helmfile
&& curl -L https://github.com/mikefarah/yq/releases/download/v4.22.1/yq_linux_amd64 -o /usr/local/bin/yq
&& chmod 0755 /usr/local/bin/yq
&& curl -L https://releases.jfrog.io/artifactory/jfrog-cli/v2-jf/2.13.0/jfrog-cli-linux-amd64/jf -o /usr/local/bin/jf
&& chmod 0755 /usr/local/bin/jf
&& su -s /bin/bash buildkite -c '/usr/local/bin/helm init --client-only'
&& su -s /bin/bash buildkite -c '/usr/local/bin/helm repo add helm-virtual https://artifactorydocker.service.csnzoo.com/artifactory/helm-virtual'
&& curl -L https://github.com/databus23/helm-diff/releases/download/v2.11.0%2B5/helm-diff-linux.tgz | tar xzv -C /home/buildkite/.helm/plugins
&& mkdir -p /home/buildkite/.local/share/helm/plugins
&& curl -L https://artifactorybase.service.csnzoo.com/artifactory/helm-plugins/diff/diff-v3.4.2.tgz | tar xvz -C /home/buildkite/.local/share/helm/plugins/
&& curl -L https://artifactorybase.service.csnzoo.com/artifactory/helm-plugins/2to3/2to3-v0.10.0.tgz | tar xvz -C /home/buildkite/.local/share/helm/plugins/
&& curl -L https://github.com/argoproj/argo-cd/releases/download/${argocd_ver}/argocd-linux-amd64 -o /usr/local/bin/argocd && chmod +x /usr/local/bin/argocd
&& chown -R buildkite: /home/buildkite
&& curl -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${kustomize_ver}/kustomize_${kustomize_ver}_linux_amd64.tar.gz -o kustomize.tar.gz && tar xvzf kustomize.tar.gz kustomize -C /usr/local/bin/
&& chmod 0755 /usr/local/bin/kustomize \

Install gcloud SDK

&& apk add --no-cache curl bash python3
&& curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-426.0.0-linux-x86_64.tar.gz
&& tar zxvf google-cloud-sdk-426.0.0-linux-x86_64.tar.gz -C /usr/local
&& chmod -R 755 /usr/local/google-cloud-sdk
&& /usr/local/google-cloud-sdk/install.sh --quiet
&& rm google-cloud-sdk-426.0.0-linux-x86_64.tar.gz
&& ln -s /usr/local/google-cloud-sdk/bin/gcloud /usr/bin/gcloud
&& ln -s /usr/local/google-cloud-sdk/bin/gsutil /usr/bin/gsutil
&& ln -s /usr/local/google-cloud-sdk/bin/bq /usr/bin/bq \

Install beta components

&& /usr/local/google-cloud-sdk/bin/gcloud components install beta --quiet
#Install gcloud-auth-plugin
&& /usr/local/google-cloud-sdk/bin/gcloud components install gke-gcloud-auth-plugin --quiet
&& echo "export USE_GKE_GCLOUD_AUTH_PLUGIN=True" >> /etc/profile
&& echo "Done"

#Install garden
RUN curl -LO https://artifactorybase.service.csnzoo.com/artifactory/garden_vrtsnc/garden/garden/wf/prod/v0.13.54/dist/garden-0.13.54-alpine-amd64.tar.gz &&
mkdir -p /tmp/garden-install &&
tar -xzf garden-0.13.54-alpine-amd64.tar.gz -C /tmp/garden-install &&
rm garden-0.13.54-alpine-amd64.tar.gz &&
mkdir -p /home/buildkite/.garden/bin &&
cp -r /tmp/garden-install/alpine-amd64/* /home/buildkite/.garden/bin &&
ln -s /home/buildkite/.garden/bin/garden /usr/local/bin/garden
&& chown -R buildkite: /home/buildkite
&& echo "Done"

Installl go

RUN curl -O https://dl.google.com/go/go1.21.9.linux-amd64.tar.gz
&& tar zxvf go1.21.9.linux-amd64.tar.gz -C /usr/local \
&& mkdir -p /usr/local/go/src/configmap
&& chmod o+rwx /usr/local/go/src/configmap
&& echo "Done"

RUN apk update && apk add libstdc++6

USER 1001

RUN mkdir -p /tmp/garden_init
COPY files/garden.base.yaml /tmp/garden_init/garden.yaml

RUN cd /tmp/garden_init
&& git init && garden util fetch-tools

Wanted to ask if we are using any particular version of node which may be causing this error? Can you help use herer

msharma38 · 2025-03-14T15:11:39Z

@stefreak I have updated the garden version to 0.13.54 but when i am running garden deploy command from buildkite getting below error

Error:-
Command "/home/buildkite/.garden/tools/mutagen/722d0c1941ba875e/mutagen sync create /home/buildkite/builds/k8s-garden-buildkite-runners-wf-buildkite-agent-c7b58899f-6hcgg-1/wayfair/garden-operation-pipeline/wf-garden-buildkite-workflows/target_repo/.garden/build/simple-golang eyJrdWJlY3RsUGF0aCI6Ii9ob21lL2J1aWxka2l0ZS8uZ2FyZGVuL3Rvb2xzL2t1YmVjdGwvNjY2ZTQ3MWEwYmI3ZWFmMy9rdWJlcm5ldGVzL2NsaWVudC9iaW4va3ViZWN0bCIsImt1YmVjdGxBcmdzIjpbImV4ZWMiLCItaSIsIi0tY29udGV4dD1na2Vfd2YtZ2NwLXVzLXBsYXQtazhzLWRldl91cy1jZW50cmFsMV9wbGF0LWRldi1jMS1kc20xIiwiLS1uYW1lc3BhY2U9ZXBoZW1lcmFsLWJrLWYyYjQ0OWY1OWFhMjYyY2MiLCItLWNvbnRhaW5lciIsInV0aWwiLCJEZXBsb3ltZW50L2dhcmRlbi11dGlsIiwiLS0iLCIvLmdhcmRlbi9tdXRhZ2VuLWFnZW50Iiwic3luY2hyb25pemVyIl19:/data/d48f136e-0c2e-4afa-9779-4979c4ee62a0/simple-golang --name k8s--build-sync--remote--ephemeral-bk-f2b449f59aa262cc--simple-golang--u9stugxb --sync-mode one-way-replica --default-file-mode 0644 --default-directory-mode 0755" failed with code 1:

Error: unable to connect to beta: unable to connect to endpoint: unable to dial agent endpoint: version handshake error: version mismatch

Here's the full output:

Creating session...
Connecting to agent (POSIX)...
Error: unable to connect to beta: unable to connect to endpoint: unable to dial agent endpoint: version handshake error: version mismatch

msharma38 added the feature request label Feb 24, 2025

vvagaytsev closed this as completed Feb 25, 2025

stefreak reopened this Feb 26, 2025

10ko assigned twelvemo Feb 27, 2025

stefreak assigned 10ko and unassigned twelvemo Mar 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE]: Use Internal k8sUtil Image #6877

[FEATURE]: Use Internal k8sUtil Image #6877

msharma38 commented Feb 24, 2025

vvagaytsev commented Feb 25, 2025 •

edited

Loading

vvagaytsev commented Feb 25, 2025

msharma38 commented Feb 26, 2025

stefreak commented Feb 26, 2025

stefreak commented Feb 26, 2025

msharma38 commented Feb 27, 2025

msharma38 commented Mar 12, 2025

stefreak commented Mar 12, 2025

msharma38 commented Mar 12, 2025 •

edited by stefreak

Loading

msharma38 commented Mar 12, 2025

stefreak commented Mar 13, 2025 •

edited

Loading

msharma38 commented Mar 13, 2025

stefreak commented Mar 13, 2025

stefreak commented Mar 13, 2025

msharma38 commented Mar 13, 2025

msharma38 commented Mar 14, 2025 •

edited

Loading

[FEATURE]: Use Internal k8sUtil Image #6877

[FEATURE]: Use Internal k8sUtil Image #6877

Comments

msharma38 commented Feb 24, 2025

Feature Request

What should the user be able to do?

Why do they want to do this? What problem does it solve?

Suggested Implementation(s)

How important is this feature for you/your team?

vvagaytsev commented Feb 25, 2025 • edited Loading

vvagaytsev commented Feb 25, 2025

msharma38 commented Feb 26, 2025

stefreak commented Feb 26, 2025

stefreak commented Feb 26, 2025

msharma38 commented Feb 27, 2025

msharma38 commented Mar 12, 2025

[k8s-buildkite-agent stage-1 9/9] RUN cd /tmp/garden_init && git init && garden util fetch-tools:

stefreak commented Mar 12, 2025

msharma38 commented Mar 12, 2025 • edited by stefreak Loading

msharma38 commented Mar 12, 2025

stefreak commented Mar 13, 2025 • edited Loading

msharma38 commented Mar 13, 2025

stefreak commented Mar 13, 2025

stefreak commented Mar 13, 2025

msharma38 commented Mar 13, 2025

Install gcloud SDK

Install beta components

Installl go

msharma38 commented Mar 14, 2025 • edited Loading

vvagaytsev commented Feb 25, 2025 •

edited

Loading

msharma38 commented Mar 12, 2025 •

edited by stefreak

Loading

stefreak commented Mar 13, 2025 •

edited

Loading

msharma38 commented Mar 14, 2025 •

edited

Loading