ci: camunda docker CI #19623

houssain-barouni · 2024-06-21T09:02:38Z

Description

add camunda docker image build and publication in the GH workflow:
1- .github/workflows/camunda-platform-release.yml
2- .github/workflows/ci.yml

docker-checks job:
- triggered when GHA or java or frontend or Docker changes
- builds a camunda/camunda image and pushs it to a local registry
- verifies the docker image labels against a golden file
- runs CamundaDockerIT integration test (now it is part of dist module but can be moved to qa module that @Zelldon is creating). This test starts an ES image and the freshly built camunda/camunda image. It checks the actuator health endpoint response
deploy-camunda-docker-snapshot job
- triggered when branch is main and the check-results is passing
- creates and publishes SNAPSHOT image to DockerHub

Related issues

relates to #18670

github-actions · 2024-06-21T09:07:35Z

Operate Opensearch ITs Results

182 tests ±0 182 ✅ ±0 1m 4s ⏱️ -10s
19 suites ±0 0 💤 ±0
19 files ±0 0 ❌ ±0

Results for commit 61eb54b. ± Comparison against base commit 4f5510b.

♻️ This comment has been updated with latest results.

github-actions · 2024-06-21T09:11:25Z

Operate Unit Tests Results

308 tests ±0 307 ✅ ±0 6m 28s ⏱️ -13s
51 suites ±0 1 💤 ±0
51 files ±0 0 ❌ ±0

Results for commit 61eb54b. ± Comparison against base commit 4f5510b.

♻️ This comment has been updated with latest results.

github-actions · 2024-06-21T09:15:04Z

Operate Integration Tests Results

535 tests ±0 533 ✅ ±0 8m 36s ⏱️ +21s
89 suites ±0 2 💤 ±0
89 files ±0 0 ❌ ±0

Results for commit 61eb54b. ± Comparison against base commit 4f5510b.

♻️ This comment has been updated with latest results.

github-actions · 2024-06-21T09:46:48Z

Tasklist Test Results

146 files ± 0 146 suites ±0 1h 27m 45s ⏱️ - 12m 0s
565 tests ± 0 560 ✅ ± 0 5 💤 ±0 0 ❌ ±0
533 runs - 30 528 ✅ - 30 5 💤 ±0 0 ❌ ±0

Results for commit 61eb54b. ± Comparison against base commit 4f5510b.

♻️ This comment has been updated with latest results.

.github/actions/paths-filter/action.yml

cmur2 · 2024-06-25T10:54:52Z

.github/workflows/ci.yml

+ docker-checks:
+ if: needs.detect-changes.outputs.docker-ci == 'true'
+ needs: [ detect-changes ]
+ runs-on: ubuntu-latest


Please add a timeout-minutes variable to limit execution time to what is conservatively needed based on observation how long it usually runs.

the first successful run took ~15 minutes https://github.com/camunda/camunda/actions/runs/9664124006/job/26658087425?pr=19623
I will put 25 minutes timeout for now, which is less than the timeout of the concurrent job java-unit-tests (30 min)

The java-unit-tests job also doesn't need the 30 minutes, it is closer to 15 the last time I checked.

I also want to keep runtimes low so 15 minutes is already at the upper bound (I put some new guidelines in https://github.com/camunda/camunda/wiki/CI-&-Automation#workflow-inclusion-criteria ) - we have to make sure to not let it grow, otherwise parallelization/caching/bigger runners need to be considered.

.github/workflows/ci.yml

Zelldon

Thanks @houssain-barouni I have some questions and comments, lets clarify them first before we move on. We can do this also sync if you like.

Zelldon · 2024-06-26T07:19:14Z

.ci/docker/test/camunda-docker-labels.golden.json

@@ -0,0 +1,21 @@
+{
+ "io.k8s.description": "Workflow engine for microservice orchestration",


❓ I guess these labels are still need to be updated?

right, it is being discussed here #19495 (comment)
if you have any suggestions, please let me know :)
otherwise I will reach out to #ask-devrel

updated some of the labels here 514a4c8
after this discussion https://camunda.slack.com/archives/C026U8GBNSW/p1719493763579659

.github/actions/paths-filter/action.yml

Zelldon · 2024-06-26T07:24:13Z

.github/actions/build-frontend/action.yml

@@ -0,0 +1,31 @@
+name: Build Frontend


❓ Didn't we had this already as a workflow or action before? 🤔 Can we use also in other places then?

we have a dedicated one to Tasklist https://github.com/camunda/camunda/blob/main/.github/actions/build-tasklist-fe/action.yml
@pedesen is also currently working on using GHA to build operate frontend #19085. I will get in touch with him to avoid overlapping work

.github/workflows/ci.yml

Zelldon · 2024-06-26T07:32:43Z

.github/workflows/ci.yml

+ deploy-camunda-docker-snapshot:
+ name: Deploy snapshot Camunda Docker image
+ needs: [ detect-changes, check-results ]
+ runs-on: ubuntu-latest


❓ 🔧 Could we reuse the image from before and just retag it?

I am afraid we can't, this is a diffrent Job and can be run by a different worker
the image built in docker-checks job is pushed to a local registery, that must be killed after the job is completed

We could maybe via caching or using job artifacts I think: https://github.com/camunda/github-actions-recipes#share-data-between-jobs-via-actionsupload-artifact-and-actionsdownload-artifact-usages-in-camunda

dist/src/test/java/io/camunda/application/CamundaDockerIT.java

Zelldon

LGTM thanks 🙇🏼

dist/src/test/java/io/camunda/application/CamundaDockerIT.java

Zelldon · 2024-06-27T09:19:39Z

dist/src/test/java/io/camunda/application/CamundaDockerIT.java

+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+
+@EnabledIfSystemProperty(named = "camunda.docker.test.enabled", matches = "true")
+public class CamundaDockerIT {


❓ Root QA module is now available on main, do you want to move it now?

yes, I have the same idea. I am planning to do it in a a new pull request.
I was planning to create a new module under QA module, dedicated to this kind of tests, with the minimum possible of dependencies: testContainers, jUnit5, apache http5...
we don't really need any dependency to the applicative code, and it should be easy to build and run tests. WDYT @Zelldon ?

Separate PR is fine by me. Extra module not sure whether this is necessary. I think you can just put it into the integration-tests module in a different package.

This test is not run within integration tests suite, but only on newly built camunda docker image (conditional on camunda.docker.test.enabled property). When we need to run this test only, we don't really need to do a full project build or import dozens of applicative snapshot artifacts that we don't use in the test.

github-actions bot added component/build-pipeline component/zeebe Related to the Zeebe component/team component/operate Related to the Operate component/team component/tasklist Related to the Tasklist component/team labels Jun 21, 2024

houssain-barouni force-pushed the camunda_docker_ci branch 2 times, most recently from a3c812a to b558919 Compare June 21, 2024 09:24

houssain-barouni force-pushed the camunda_docker_ci branch 9 times, most recently from a6b15ff to b91b5c2 Compare June 25, 2024 09:47

cmur2 requested changes Jun 25, 2024

View reviewed changes

houssain-barouni force-pushed the camunda_docker_ci branch 5 times, most recently from cf4559a to 70066b8 Compare June 25, 2024 14:14

ci: camunda docker CI

ec1c2c8

houssain-barouni force-pushed the camunda_docker_ci branch from 70066b8 to ec1c2c8 Compare June 25, 2024 15:04

houssain-barouni added 3 commits June 25, 2024 18:38

Merge branch 'camunda_docker_file' into camunda_docker_ci

6a72b6d

refactor: code review changes

63cbc9b

ci: use REGISTRY_HUB_DOCKER_COM_PSW_READ_ONLY for docker checks

3ea015f

houssain-barouni marked this pull request as ready for review June 26, 2024 06:42

houssain-barouni requested review from romansmirnov, Zelldon and cmur2 June 26, 2024 06:42

Zelldon reviewed Jun 26, 2024

View reviewed changes

refactor: code review changes: 2nd round

ecd6e58

Zelldon approved these changes Jun 27, 2024

View reviewed changes

refactor: add identity client build and other changes

61eb54b

houssain-barouni force-pushed the camunda_docker_ci branch from 65a057d to 61eb54b Compare June 27, 2024 12:21

Base automatically changed from camunda_docker_file to main June 27, 2024 12:59

houssain-barouni added 2 commits June 27, 2024 17:34

Merge remote-tracking branch 'origin/main' into camunda_docker_ci

9ab8485

refactor: update camunda docker labels

514a4c8

github-actions bot added the component/identity Related to the Identity component/team label Jun 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: camunda docker CI #19623

ci: camunda docker CI #19623

houssain-barouni commented Jun 21, 2024 •

edited

Loading

github-actions bot commented Jun 21, 2024 •

edited

Loading

github-actions bot commented Jun 21, 2024 •

edited

Loading

github-actions bot commented Jun 21, 2024 •

edited

Loading

github-actions bot commented Jun 21, 2024 •

edited

Loading

cmur2 Jun 25, 2024

houssain-barouni Jun 25, 2024 •

edited

Loading

cmur2 Jun 26, 2024

Zelldon left a comment

Zelldon Jun 26, 2024

houssain-barouni Jun 26, 2024 •

edited

Loading

houssain-barouni Jun 27, 2024

Zelldon Jun 26, 2024

houssain-barouni Jun 26, 2024 •

edited

Loading

Zelldon Jun 26, 2024

houssain-barouni Jun 26, 2024

cmur2 Jun 26, 2024

Zelldon left a comment

Zelldon Jun 27, 2024

houssain-barouni Jun 27, 2024

Zelldon Jun 27, 2024

houssain-barouni Jun 27, 2024 •

edited

Loading

		@@ -0,0 +1,21 @@
		{
		"io.k8s.description": "Workflow engine for microservice orchestration",

ci: camunda docker CI #19623

Are you sure you want to change the base?

ci: camunda docker CI #19623

Conversation

houssain-barouni commented Jun 21, 2024 • edited Loading

Description

Related issues

github-actions bot commented Jun 21, 2024 • edited Loading

Operate Opensearch ITs Results

github-actions bot commented Jun 21, 2024 • edited Loading

Operate Unit Tests Results

github-actions bot commented Jun 21, 2024 • edited Loading

Operate Integration Tests Results

github-actions bot commented Jun 21, 2024 • edited Loading

Tasklist Test Results

Choose a reason for hiding this comment

houssain-barouni Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zelldon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

houssain-barouni Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

houssain-barouni Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zelldon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

houssain-barouni Jun 27, 2024 • edited Loading

Choose a reason for hiding this comment

houssain-barouni commented Jun 21, 2024 •

edited

Loading

github-actions bot commented Jun 21, 2024 •

edited

Loading

github-actions bot commented Jun 21, 2024 •

edited

Loading

github-actions bot commented Jun 21, 2024 •

edited

Loading

github-actions bot commented Jun 21, 2024 •

edited

Loading

houssain-barouni Jun 25, 2024 •

edited

Loading

houssain-barouni Jun 26, 2024 •

edited

Loading

houssain-barouni Jun 26, 2024 •

edited

Loading

houssain-barouni Jun 27, 2024 •

edited

Loading