Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: update YQ everywhere and deploy-renku wants split values files. #60

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

chore: update YQ everywhere and deploy-renku wants split values files. #60

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions deploy-renku/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
FROM alpine/k8s:1.23.17
FROM alpine/k8s:1.24.16

# install dependencies
COPY requirements.txt /
RUN apk add --no-cache python3 docker jq && \
wget -O /usr/bin/yq "https://github.com/mikefarah/yq/releases/download/3.1.1/yq_linux_amd64" && \
chmod a+x /usr/bin/yq && \
RUN apk add python3 docker jq yq && \
pip3 install -r /requirements.txt

COPY deploy-dev-renku.py entrypoint.sh /
Expand Down
18 changes: 12 additions & 6 deletions deploy-renku/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,14 @@ echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
# set up kube context and values file
echo "$RENKUBOT_KUBECONFIG" > "$KUBECONFIG" && chmod 400 "$KUBECONFIG"

# set up the values file
printf "%s" "$RENKU_VALUES" | sed "s/<replace>/${RENKU_RELEASE}/" > $RENKU_VALUES_FILE
export RENKU_VALUES_FILE="/renku-values.yaml"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is already a RENKU_VALUES_FILE variable defined (or at least that is what the README says) and that contains the values.yaml file that comes with the helm chart. But here this is something else I think? Is this supposed to contain the non-secret CI values?


# merge the secret and clear value files
printf "%s" "$RENKU_SECRET_VALUES" > $RENKU_VALUES_FILE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to update the README.md for these changes accordingly. According to the README the secret values are called just RENKU_VALUES.

yq eval-all --inplace '. as $item ireduce ({}; . * $item )' $RENKU_VALUES_FILE $RENKU_CLEAR_VALUES_FILE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better to let helm take care of this if we simply pass the files on to the helm command in the python script that does helm upgrade --install. I.e. we can do helm install -f non-secret.values.yaml -f secret-values.yaml --set extra-values-from-comment. Then we dont have to do this yq magic stuff. Helm already handles this.


# replace the release name
sed --in-place "s/<replace>/${RENKU_RELEASE}/" $RENKU_VALUES_FILE

# register the GitLab app
if test -n "$GITLAB_TOKEN" ; then
Expand All @@ -22,12 +28,12 @@ if test -n "$GITLAB_TOKEN" ; then
--data "name=${RENKU_RELEASE}" \
--data "redirect_uri=https://${RENKU_RELEASE}.dev.renku.ch/auth/realms/Renku/broker/dev-renku/endpoint https://${RENKU_RELEASE}.dev.renku.ch/api/auth/gitlab/token" \
--data "scopes=api read_user read_repository read_registry openid")
APP_ID=$(echo $gitlab_app | jq -r '.application_id')
APP_SECRET=$(echo $gitlab_app | jq -r '.secret')
export APP_ID=$(echo $gitlab_app | jq -r '.application_id')
export APP_SECRET=$(echo $gitlab_app | jq -r '.secret')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am pretty sure we don't need the export here and in front of APP_ID and for RENKU_VALUES_FILE. The variables will be available within the script even without export. So no need to do it.


# gateway gitlab app/secret
yq w -i $RENKU_VALUES_FILE "gateway.gitlabClientId" "$APP_ID"
yq w -i $RENKU_VALUES_FILE "gateway.gitlabClientSecret" "$APP_SECRET"
yq eval --inplace '.gateway.gitlabClientId = strenv(APP_ID)' $RENKU_VALUES_FILE
yq eval --inplace '.gateway.gitlabClientSecret = strenv(APP_SECRET)' $RENKU_VALUES_FILE
fi

# create namespace and ignore error in case it already exists
Expand Down
5 changes: 1 addition & 4 deletions publish-chartpress-images/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
FROM docker:24.0.6-git

# install dependencies
# Note: Chartpress 0.7.0 is the latest version compatible until https://github.com/jupyterhub/chartpress/issues/118 is fixed
RUN apk add python3 py-pip py3-ruamel.yaml && \
wget -O /usr/bin/yq "https://github.com/mikefarah/yq/releases/download/3.1.1/yq_linux_amd64" && \
chmod 0755 /usr/bin/yq && \
RUN apk add python3 py-pip && \
pip3 install -U pip chartpress==2.1.0 && \
wget -O /tmp/helm.tar.gz "https://get.helm.sh/helm-v3.5.2-linux-amd64.tar.gz" && \
tar -xf /tmp/helm.tar.gz --strip-components=1 -C /usr/bin/ && \
Expand Down
9 changes: 3 additions & 6 deletions rollout-renku-version/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
FROM python:3.7-alpine3.10
FROM alpine:3.18

# install dependencies
RUN apk add --no-cache git bash && \
apk add --no-cache --virtual .build-deps gcc g++ make && \
wget -O /usr/bin/yq "https://github.com/mikefarah/yq/releases/download/v4.27.5/yq_linux_amd64" && \
chmod a+x /usr/bin/yq && \
apk del .build-deps
RUN apk update && \
apk add --no-cache git bash yq

COPY rollout-renku-deployment.sh /
RUN chmod +x /rollout-renku-deployment.sh
Expand Down
2 changes: 1 addition & 1 deletion rollout-renku-version/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Action for updating component version

This is a docker action that will update the Renku version (rollout) in the deployments managed by terraform.
This is a docker action that will update the Renku version (rollout) in the deployments managed by Terraform.

## Sample usage

Expand Down
4 changes: 2 additions & 2 deletions rollout-renku-version/rollout-renku-deployment.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
set -xe
set -e

if [ -z "$GITHUB_TOKEN" ]
then
Expand Down Expand Up @@ -47,7 +47,7 @@ do
# update renku version and push
git checkout -b auto-update/${CHART_NAME}-${CHART_VERSION}-${cluster} ${UPSTREAM_BRANCH}

yq -i '.spec.chart.spec.version = strenv(CHART_VERSION)' $cluster_dir/main/charts/renku.yaml
yq eval --inplace '.spec.chart.spec.version = strenv(CHART_VERSION)' $cluster_dir/main/charts/renku.yaml
sed -i "/Renku version/c\ ### Renku version $CHART_VERSION ($DATE)" $cluster_dir/main/charts/renku.yaml
sed -i "/Release Notes/c\ See the [Release Notes](https://github.com/${GITHUB_REPOSITORY}/releases/tag/$CHART_VERSION)" $cluster_dir/main/charts/renku.yaml

Expand Down
13 changes: 6 additions & 7 deletions update-component-version/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
FROM python:3.7-alpine3.10
FROM python:3.9-alpine3.18

# install dependencies
RUN apk add --no-cache git bash && \
apk add --no-cache --virtual .build-deps gcc g++ make && \
wget -O /usr/bin/yq "https://github.com/mikefarah/yq/releases/download/3.1.1/yq_linux_amd64" && \
chmod a+x /usr/bin/yq && \
pip install -U pip chartpress==2.1.0 six==1.16.0 && \
apk del .build-deps
RUN apk add --no-cache git bash yq && \
pip install -U pip \
chartpress==2.1.0 \
'ruamel.yaml<0.17.10' \
'ruamel.yaml.clib<0.2.4'

COPY update-upstream.sh /
ENTRYPOINT [ "/update-upstream.sh" ]
6 changes: 3 additions & 3 deletions update-component-version/update-upstream.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
set -xe
set -e

if [ -z "$GITHUB_TOKEN" ]
then
Expand All @@ -21,7 +21,7 @@ COMPONENT_NAME=${COMPONENT_NAME:=$(echo $GITHUB_REPOSITORY | cut -d/ -f2)}

# build this chart to get the version
chartpress --skip-build $COMPONENT_TAG
COMPONENT_VERSION=$(yq r helm-chart/${COMPONENT_NAME}/Chart.yaml version)
COMPONENT_VERSION=$(yq eval '.version' helm-chart/${COMPONENT_NAME}/Chart.yaml)

git clone --depth=1 --branch=${UPSTREAM_BRANCH} https://${GITHUB_TOKEN}@github.com/${UPSTREAM_REPO} upstream-repo

Expand All @@ -34,7 +34,7 @@ git config --global user.name "$GIT_USER"

# update the chart requirements and push
git checkout -b auto-update/${COMPONENT_NAME}-${COMPONENT_VERSION} ${UPSTREAM_BRANCH}
yq m -x -i helm-chart/renku/values.yaml ../helm-chart/${COMPONENT_NAME}/values.yaml
yq eval-all --inplace '. as $item ireduce ({}; . * $item )' helm-chart/renku/values.yaml ../helm-chart/${COMPONENT_NAME}/values.yaml

git add helm-chart/renku/values.yaml
git commit -m "chore: updating ${COMPONENT_NAME} version to ${COMPONENT_VERSION}"
Expand Down
2 changes: 1 addition & 1 deletion update-renku-actions/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM alpine:3.15
FROM alpine:3.18

RUN apk add --no-cache yq
COPY entrypoint.sh /
Expand Down