Improve GitHub Actions #329

itskingori · 2024-10-16T17:24:58Z

Thought I'd help improve things a little bit, basically:

Consolidate all test jobs into a "Test" workflow and all release jobs into a "Release" workflow.
Configure Docker image release job to push to both GitHub Container Registry and Docker Hub.
Add GitHub release of the application and have the Helm Chart packaged and uploaded as assets of the release.

Reviewing

Follow the commits for an easier review, each commit is an idea.

Motivations

I wanted linux/arm64 support on Docker Hub but saw that it's just linux/amd64 (though it's on GItHub Container Registry). Found these two related issues & PRs trying to understand the current status:

Related Automatically push to ghcr and build for ARM #327
Closes multi platform - ARM64 release #171
Closes Add version tagged Docker image #35

Initially, I found the current organisation of GitHub Actions across multiple different workflows confusing because it wasn't clear which ones one is test or release jobs.

My belief is that most jobs can be categorised along those two broad categories. So this PR is a very subjective view of how I think it should be i.e. different triggers for test and release jobs on just two workflows then those that trigger the relevant jobs.

Preview

References

danihodovic · 2024-10-16T18:03:09Z

cc @adinhodovic

…hub-release

itskingori

I've updated the description to give more context. I am willing to make modifications and explain any part that isn't clear.

itskingori

Some commentary for even more context ... 💬

itskingori · 2024-10-16T18:32:53Z

.github/workflows/release.yml

+  github:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Create Release
+        uses: actions/create-release@v1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          tag_name: ${{ github.ref }}
+          release_name: Version ${{ github.ref }}
+          draft: false
+          prerelease: false


This is new. It creates GitHub release after a tag is created. To note, the other release jobs are dependent on this on since they have needs: github set.

itskingori · 2024-10-16T18:34:16Z

.github/workflows/release.yml

+          username: ${{ secrets.DOCKER_HUB_USERNAME }}
+          password: ${{ secrets.DOCKER_HUB_TOKEN }}


You'll need to set DOCKER_HUB_USERNAME and DOCKER_HUB_TOKEN as GitHub Actions secrets.

itskingori · 2024-10-16T18:37:34Z

.github/workflows/release.yml

+      - name: Build, tag, and push image to Docker Hub
+        uses: docker/build-push-action@v6
+        with:
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+          context: .
+          labels: ${{ steps.metadata.outputs.labels }}
+          platforms: linux/amd64,linux/arm64
+          push: true
+          tags: ${{ steps.metadata.outputs.tags }}


The idea here is the metadata action will create tags for the multiple images set here and since we're authenticated into both registries (GitHub Container Registry & Docker Hub) at this point, both pushes should work.

itskingori · 2024-10-16T18:39:14Z

.github/workflows/release.yml

+on:
+  push:
+    tags:
+      - '[0-9]+.[0-9]+.[0-9]+'


To note, I removed the preceding v for better flexibility in the jobs. This way, in the downstream jobs you can manually prefix where applicable rather than assume it will be everywhere (which it isn't e.g. Docker image tags).

itskingori · 2024-10-16T18:41:23Z

.github/workflows/release.yml

+      - name: Update Description On Docker Hub Description
+        uses: peter-evans/dockerhub-description@v4
+        with:
+          username: ${{ secrets.DOCKER_HUB_USERNAME }}
+          password: ${{ secrets.DOCKER_HUB_TOKEN }}
+          repository: ${{ env.IMAGE }}


Automates the overview on Docker Hub, which is currently empty ...

itskingori · 2024-10-16T18:43:14Z

.github/workflows/test.yml

+name: Test
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:


With all these test jobs as part of one workflow it will look great. Obviously right now the workflow isn't approved so it isn't running but it should look something like this ...

itskingori

Updated PR description and updated release workflow to have the Helm Chart packaged and uploaded as an assets of the release instead of having two releases (one for the app and another for the chart).

Warning

It's worth noting that the release is untested as it only runs when a tag happens. I've done my best to review the logic and I am willing to fix any issues if it fails.

itskingori · 2024-10-17T02:51:46Z

.github/workflows/release.yml

+      - name: Get Chart Version
+        id: chart-version
+        run: echo "::set-output name=version::$(grep 'version:' ./charts/celery-exporter/Chart.yaml | cut -d ' ' -f 2)"
+      - name: Package Helm chart
+        run: helm package ./charts/celery-exporter --app-version=${{ github.ref_name }}
+      - name: Upload To Github Release
+        uses: softprops/action-gh-release@v2
+        with:
+          files: |
+            celery-exporter-${{ steps.chart-version.outputs.version }}.tgz
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}


Instead of creating a release for the Helm Chart like this ...

celery-exporter/.github/workflows/helm-release.yml

Lines 31 to 35 in db019b8

- name: Run chart-releaser

uses: helm/[email protected]

env:

CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"

CR_RELEASE_NAME_TEMPLATE: "{{ .Name }}-chart-{{ .Version }}"

Have it uploaded as an asset to the GitHub release (on tag), because each time we tag we'll get two releases which can be cleaner, here's what we have now ...

itskingori · 2024-10-17T02:53:19Z

.github/workflows/release.yml

+          version: v3.14.4
+      - name: Get Chart Version
+        id: chart-version
+        run: echo "::set-output name=version::$(grep 'version:' ./charts/celery-exporter/Chart.yaml | cut -d ' ' -f 2)"


Basic test that this works ...

$ ls -l ./charts/* total 24 -rw-r--r-- 1 Kingori staff 333 Oct 17 04:22 Chart.yaml -rw-r--r-- 1 Kingori staff 2790 Oct 16 18:57 README.md drwxr-xr-x 3 Kingori staff 96 Oct 16 18:57 ci drwxr-xr-x 10 Kingori staff 320 Oct 16 18:57 templates -rw-r--r-- 1 Kingori staff 2640 Oct 16 18:57 values.yaml $ grep 'version:' ./charts/celery-exporter/Chart.yaml | cut -d ' ' -f 2 0.7.0

itskingori · 2024-10-17T02:54:28Z

.github/workflows/release.yml

+        id: chart-version
+        run: echo "::set-output name=version::$(grep 'version:' ./charts/celery-exporter/Chart.yaml | cut -d ' ' -f 2)"
+      - name: Package Helm chart
+        run: helm package ./charts/celery-exporter --app-version=${{ github.ref_name }}


Local test that this works ...

$ helm package ./charts/celery-exporter --app-version=0.10.13 Successfully packaged chart and saved it to: /Users/Kingori/Repositories/itskingori/celery-exporter/celery-exporter-0.7.0.tgz

itskingori · 2024-10-17T02:59:48Z

.github/workflows/release.yml

+        uses: softprops/action-gh-release@v2
+        with:
+          files: |
+            celery-exporter-${{ steps.chart-version.outputs.version }}.tgz


Should be available after packaging ...

$ helm package ./charts/celery-exporter --app-version=0.10.13 Successfully packaged chart and saved it to: /Users/Kingori/Repositories/itskingori/celery-exporter/celery-exporter-0.7.0.tgz $ ls -l celery-exporter-0.7.0.tgz -rw-r--r-- 1 Kingori staff 4981 Oct 17 04:53 celery-exporter-0.7.0.tgz

itskingori · 2024-10-17T03:01:59Z

charts/celery-exporter/Chart.yaml

-
 version: 0.7.0
-appVersion: 0.9.2


Preferring to set the app-version dynamically i.e. with the --app-version flag available on helm package ...

$ helm package --help | grep '\-version' --app-version string set the appVersion on the chart to this version --version string set the version on the chart to this semver version

danihodovic · 2024-10-17T16:50:35Z

This will take some time to review

itskingori · 2024-10-17T18:57:10Z

No problem. I'll keep an eye out for notifications whenever you get round to looking at it.

itskingori · 2024-10-17T19:10:11Z

charts/celery-exporter/Chart.yaml

-version: 0.7.0
+version: 0.7.1


Bumped because the test/helm-chart job is failing due no version bump detected despite the removal of appVersion in b697d34 ...

zacblazic · 2024-10-21T12:59:58Z

charts/celery-exporter/Chart.yaml

@@ -12,6 +12,4 @@ sources:
 maintainers:
  - name: danihodovic
  - name: adinhodovic
-
-version: 0.7.0
-appVersion: 0.9.2


Did we intend to remove this line?

Yes, in preference to dynamically setting it as done below with the --app-version flag ...

celery-exporter/.github/workflows/release.yml

Line 97 in cf5944a

run: helm package ./charts/celery-exporter --app-version=${{ github.ref_name }}

As it should track the app version which is only determined during the actual release.

itskingori · 2024-11-19T07:54:24Z

I should have probably filed an issue to confirm if the scope of my suggestions is acceptable (and reviewable). Been waiting a bit for this but now opting to package my own image in Intellection/docker-celery-exporter as it's urgent ATM.

These ideas have been implemented in the below PRs:

danihodovic · 2024-11-19T17:47:30Z

I should have probably filed an issue to confirm if the scope of my suggestions is acceptable (and reviewable). Been waiting a bit for this but now opting to package my own image in Intellection/docker-celery-exporter as it's urgent ATM.

Sounds good. I still haven't gotten around to looking at the change as work has been hectic and I release a version of celery-exporter about once a month on average.

itskingori · 2024-11-20T04:14:10Z

I understand. My changes were also not easily digestible. I kinda struggled with needing approvals for the workflows to run each time I pushed, so I can test.

Feel free to have a look at the workflows in Intellection/docker-celery-exporter (and copy if you want, MIT licensed), they pretty much just check out the code here and do what I was aiming to do in totality (at least on the Docker build front).

Got an ARM based build finally, so I'm unblocked for now ...

$ docker buildx imagetools inspect zappi/celery-exporter:0.10.14
Name:      docker.io/zappi/celery-exporter:0.10.14
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:2226d248f957af47d47476315c8dd83d5ace3796470d96b90870bba286b00549

Manifests:
  Name:        docker.io/zappi/celery-exporter:0.10.14@sha256:1c514308647461b97d1630372842140e562e3101eaf20639c3703f61385c9804
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64

  Name:        docker.io/zappi/celery-exporter:0.10.14@sha256:400845b8bbf89fcaf4b61ca709420a0aba807964397ecaecd4c88da3369bcf51
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64

  Name:        docker.io/zappi/celery-exporter:0.10.14@sha256:0c89f89db4b97b37bc46f49580430932053f98740bdab4323d6ec86bd9cb30a6
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.digest: sha256:1c514308647461b97d1630372842140e562e3101eaf20639c3703f61385c9804
    vnd.docker.reference.type:   attestation-manifest

  Name:        docker.io/zappi/celery-exporter:0.10.14@sha256:dc4e1cdee61c8ff571a907488eada65db5d681092fb1191487025e8784e9ade3
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.digest: sha256:400845b8bbf89fcaf4b61ca709420a0aba807964397ecaecd4c88da3369bcf51
    vnd.docker.reference.type:   attestation-manifest

itskingori force-pushed the improve_github_actions branch 2 times, most recently from 498110c to 75b30c9 Compare October 16, 2024 17:56

itskingori force-pushed the improve_github_actions branch from 75b30c9 to 8106c50 Compare October 16, 2024 18:05

Add test workflow

30cbeaf

itskingori force-pushed the improve_github_actions branch from 8106c50 to d75f703 Compare October 16, 2024 18:13

itskingori added 6 commits October 16, 2024 20:18

Add release workflow

c4941f3

Move helm-test job to test workflow

d233b35

Move ci/lint job to test workflow

33b96f3

Move ci/test job to test workflow

953e956

Move helm-release/release job to release workflow

4004cfa

Combine push-image-ghcr/build-and-push-image job with release/docker-…

353ce08

…hub-release

itskingori force-pushed the improve_github_actions branch from d75f703 to 353ce08 Compare October 16, 2024 18:19

itskingori marked this pull request as ready for review October 16, 2024 18:22

itskingori commented Oct 16, 2024

View reviewed changes

itskingori added 2 commits October 17, 2024 04:42

Upload packaged Helm Chart to GitHub release's assets

b697d34

Prefer reference without the refs/tags/

ac3e1d1

itskingori commented Oct 17, 2024

View reviewed changes

This was referenced Oct 17, 2024

multi platform - ARM64 release #171

Open

Add version tagged Docker image #35

Open

Add annotations to docker build

8f88421

Bump chart to appease linter after appVersion removal

cf5944a

itskingori commented Oct 17, 2024

View reviewed changes

zacblazic reviewed Oct 21, 2024

View reviewed changes

itskingori closed this Nov 19, 2024

itskingori mentioned this pull request Nov 19, 2024

Add test and release workflows to build multi-arch Docker image Intellection/docker-celery-exporter#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve GitHub Actions #329

Improve GitHub Actions #329

itskingori commented Oct 16, 2024 •

edited

Loading

danihodovic commented Oct 16, 2024

itskingori left a comment

itskingori left a comment

itskingori Oct 16, 2024

itskingori Oct 16, 2024

itskingori Oct 16, 2024

itskingori Oct 16, 2024

itskingori Oct 16, 2024

itskingori Oct 16, 2024

itskingori left a comment

itskingori Oct 17, 2024

itskingori Oct 17, 2024

itskingori Oct 17, 2024

itskingori Oct 17, 2024

itskingori Oct 17, 2024

danihodovic commented Oct 17, 2024

itskingori commented Oct 17, 2024

itskingori Oct 17, 2024

zacblazic Oct 21, 2024

itskingori Oct 21, 2024

itskingori commented Nov 19, 2024 •

edited

Loading

danihodovic commented Nov 19, 2024

itskingori commented Nov 20, 2024

		username: ${{ secrets.DOCKER_HUB_USERNAME }}
		password: ${{ secrets.DOCKER_HUB_TOKEN }}

	- name: Run chart-releaser
	uses: helm/[email protected]
	env:
	CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
	CR_RELEASE_NAME_TEMPLATE: "{{ .Name }}-chart-{{ .Version }}"

Improve GitHub Actions #329

Improve GitHub Actions #329

Conversation

itskingori commented Oct 16, 2024 • edited Loading

Reviewing

Motivations

Preview

References

danihodovic commented Oct 16, 2024

itskingori left a comment

Choose a reason for hiding this comment

itskingori left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itskingori left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danihodovic commented Oct 17, 2024

itskingori commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itskingori commented Nov 19, 2024 • edited Loading

danihodovic commented Nov 19, 2024

itskingori commented Nov 20, 2024

itskingori commented Oct 16, 2024 •

edited

Loading

itskingori commented Nov 19, 2024 •

edited

Loading