kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

ljmotta · 2025-01-21T22:08:30Z

Closes apache/incubator-kie-issues#1114
Closes #2839

Description

This PR introduces containerized end-to-end (E2E) Playwright tests to ensure consistent and reliable test execution across environments. By default, after this PR, running the E2E tests locally and in the CI will use containers instead of running natively in the OS.

Tests were executed in the following environments:

macOS
- arm64
- Sonoma 14.6.1
- Podman Desktop v1.15
- Podman Engine 5.3.1
- Docker CLI 27.2.1
macOS
- amd64
- Sonoma 14.5
- Podman Desktop v1.16
- Podman Engine 5.3.2
- Docker CLI 274.0
Windows WSL
- amd64
- Ubuntu 22.04.3 LTS
- Docker CE CLI 25.0.2
- Docker CE Engine 25.0.2

This PR updates multiple screenshots, as the changes introduced allowed lowering the snapshot threshold from 10% to 0.1%!

Known Issues

Running the online-editor containerized test-e2e script on macOS requires starting the services in a separate terminal.
Performance: Tests may run slower on macOS arm64 due to architecture differences.
Flakiness: Retries are enabled because tests can occasionally fail without valid reasons when running in the restricted containerized environment.
Debugging tests is not currently supported but could be added in the future.
The Docker CLI is required to run the containerized tests.

Trying it out

Install the dependencies.
Build the @kie-tools/playwright-base image with the following command:

KIE_TOOLS_BUILD__buildContainerImages=true pnpm -F @kie-tools/playwright-base image:docker:build

In a package that contains Playwright e2e tests, run the following:

KIE_TOOLS_BUILD__runEndToEndTests=true pnpm test-e2e

This command will start the required services and run the tests inside the container.

Updating screenshots

To update screenshots, use the pnpm test-e2e:container:shell script to open a shell inside the container:

pnpm test-e2e:container:shell
# In the container shell, manually call the `test-e2e:run` passing the `-u` or `--update-snapshots` flag
pnpm test-e2e:run -u
# or
pnpm test-e2e:run --update-snapshots

To improve performance, you can filter which tests will run by adding the -g flag:

pnpm test-e2e:run -u -g "test name"

…ne-editor

jomarko · 2025-01-22T09:28:20Z

Thank you @ljmotta . Please, let us know this is ready for a verification. I will try on non-MAC environment.

ljmotta · 2025-01-23T10:51:29Z

@jomarko This PR can be tested for macOS and Linux. I still need to make some adjustments for native Windows (non WSL).

thiagoelg

Awesome PR @ljmotta! This will make our e2e tests more stable and concise!

I've made some comments below about the package.json scripts; let me know what you think.

packages/playwright-base/package.json

packages/online-editor/package.json

ljmotta · 2025-01-28T20:35:22Z

@tiagobento I've renamed KIE_TOOLS_BUILD__runContainerizedEndToEndTests to KIE_TOOLS_BUILD__containerizedEndToEndTests, and instead of using a new variable, I've made it a property of buildEnv.endToEndTests. I've also updated the scripts to follow the new behavior you described:

KIE_TOOLS_BUILD__runEndToEndTests=true ?
  KIE_TOOLS_BUILD__containerizedEndToEndTests=true ? 
    run_on_container : 
    run_on_host : 
  not_run

Currently, KIE Sandbox tests aren't working on macOS containers. I'm not sure why. I've tried debugging it, but I haven't found any obvious cause. The DMN and BPMN Editors are stuck in an infinite loading state and aren't fetching their respective envelopes. The shared worker is stuck in the git.init method, which calls the fs.exists method. From what I can see, the fs.exists method doesn't exist 😅.

tiagobento · 2025-01-29T17:05:33Z

Did you try serving online-editor with something like http-server, for instance, instead of the Webpack Dev Server (pnpm start)?

ljmotta · 2025-01-29T21:21:56Z

@tiagobento http-server doesn't work too. I've tested with macOS in a amd64 machine, and the online-editor Google Chrome and Chormium tests passed with Docker and Podman.

@thiagoelg gave me the idea of removing the amd64 compatibility that Docker and Podman provide and running the container natively on the arm macOS. After removing the platform specification from the image build script and docker-compose file, the online-editor Chromium tests passed🎉 (Unfortunelly doesn't exist Google Chrome for Linux arm64). Everything points to a bug in the platform compatibility.

EDIT:
I see three ways we can proceed with this PR:

Remove the platform flag, and disable all Google Chrome tests for Linux arm64.
Keep the platform (since it works for the other packages) and disable Chromium and Google Chrome tests in the online-editor package.
Make the arm64 macOS user generate an additional image for arm64 that will be used for the online-editor and without Google Chrome

tiagobento · 2025-01-31T21:47:21Z

Thank you @ljmotta. From what I understand I believe the option benefiting the larger surface area would be option 2.

Please confirm if the following statement is correct:

If we go with option 2:

Developers on macOS arm64 trying to update online-editor screenshots won't be able to do so, and will need to find an alternative setup while we figure out what's the problem there.
Windows users will need to rely on WSL to update screenshots and we won't be pursuing finding the solution for non-WSL.

If this is accurate, then you have my blessing to go with it 😛

ljmotta · 2025-01-31T22:32:31Z

Yeah, that's correct. And just to be more clear. With option 2, the macOS arm64 users will not be able to run any online-editor Chromium and Google Chrome test inside the container.

…ontainerization

thiagoelg

Awesome work @ljmotta! LGTM

ljmotta · 2025-02-11T15:34:18Z

I just double checked all screenshots. It's looking good. @tiagobento could you review the PR one more time?

tiagobento · 2025-02-11T20:16:46Z

packages/online-editor/playwright.config.ts

@@ -61,6 +62,35 @@ const customConfig = defineConfig({
      timeout: 240000,
    },
  ],
+  // Override
+  projects: process.env.KIE_TOOLS_ONLINE_EDITOR__SKIP_PLAYWRIGHT_FOR_ARM64


Not using build-env here and directly accessing an env var. This is not aligned with the repository's convention...

This env var is only used for a workaround. I didn't add it to the build-env as it runs a bash script:

"test-e2e:container:up:darwin": KIE_TOOLS_ONLINE_EDITOR__SKIP_PLAYWRIGHT_FOR_ARM64=$( [[ $(uname -p) == \"arm\" ]] ...

I'm not sure how we could add this to the env.js file...

Well, I could export the variable in the script, to use it in the playwright.config.ts... 😅

tiagobento · 2025-02-11T20:19:12Z

packages/scesim-editor/tests-e2e/__containerization__/playwright-docker-compose.yml

+  playwright:
+    platform: linux/amd64
+    container_name: kie-tools-playwright-containerization-scesim-editor
+    image: dev.local/${USER}/kie-tools-playwright-container-image:${PROJECT_VERSION}


This name is not parameterized, and the ${USER} component there seems unnecessary. WDYT about

dev.local/apache-kie-tools/playwright-e2e-tests-image?

tiagobento · 2025-02-11T20:31:40Z

packages/online-editor/playwright.config.ts

@@ -61,6 +62,35 @@ const customConfig = defineConfig({
      timeout: 240000,
    },
  ],
+  // Override
+  projects: process.env.KIE_TOOLS_ONLINE_EDITOR__SKIP_PLAYWRIGHT_FOR_ARM64


Not using build-env to access an environment variable...

tiagobento · 2025-02-11T20:32:56Z

packages/online-editor/tests-e2e/__containerization__/playwright-docker-compose.yml

+  playwright:
+    platform: linux/amd64
+    container_name: kie-tools-playwright-containerization-online-editor
+    image: dev.local/${USER}/kie-tools-playwright-container-image:${PROJECT_VERSION}


Images should have its tag version aligned with streamName, not with version.

Also the ${USER} component seems unnecessary..

WDYT about dev.local/apache-kie-tools/playwright-e2e-tests-image?

ljmotta added 21 commits April 11, 2024 21:49

initial steps to create a playwright environment with docker

c432d4b

Merge remote-tracking branch 'origin/main' into playwright-docker

6c1ac33

Add container solution to generate playwright screenshots

1d47181

run bash script after up

1a44663

Update container image and scripts

7fad5fa

Use run script os in check script, and use container on all runs

227aac0

Add containerizedEndToEndTests

1fe89ff

Improve scripts and use dynamic host

09eb7c0

Add missing cross-env

f53f4b3

Rollback operator.yml

dd1b1e4

Simplify CI

69fd7c3

package.json adjustments

1a81a4b

Simplify container file

ce08de1

Make it compatible with macOS

5963a30

Use nginx to forward host.docker.internal to localhost - missing onli…

c42ba1e

…ne-editor

Add reverser proxies for other services

fad1d53

Rollback CI

9ff8897

Writting playwright-base README

1f9945a

Oops

5cfabe0

Clean up

3b89a5b

Update README and enable retries

f4cc454

ljmotta added area:dependencies Pull requests that update a dependency file area:tests labels Jan 21, 2025

ljmotta self-assigned this Jan 21, 2025

ljmotta requested a review from tiagobento as a code owner January 21, 2025 22:08

Remove unused dependency

5d7a64e

ljmotta added the pr: wip PR is still under development label Jan 21, 2025

thiagoelg reviewed Jan 23, 2025

View reviewed changes

packages/playwright-base/package.json Outdated Show resolved Hide resolved

packages/online-editor/package.json Outdated Show resolved Hide resolved

Fix start script

c7d0b2e

ljmotta added 5 commits February 3, 2025 12:06

Add condition for macOS arm64

99f2885

Merge remote-tracking branch 'origin/main' into no-issue-playwright-c…

d5dfdb6

…ontainerization

Update pnpm-lock

4e99b70

Change image to ubuntu22

5c53208

Add missing fonts, bump up expect time and decrease screenshot threshold

9c16001

ljmotta force-pushed the no-issue-playwright-containerization branch from 4a73e71 to d393657 Compare February 5, 2025 02:44

yesamer mentioned this pull request Feb 5, 2025

kie-issues#970: Test Scenario Editor: Integrate the @kie-tools/scesim-editor component with the DMN Marshaller #2887

Merged

ljmotta force-pushed the no-issue-playwright-containerization branch 3 times, most recently from c412e61 to 0640d15 Compare February 6, 2025 22:20

ljmotta added 2 commits February 10, 2025 22:19

Use docker-compose override, and fix ci problems

c07f3e9

Update snapshots

5539cbf

ljmotta force-pushed the no-issue-playwright-containerization branch from 0a55a93 to 5539cbf Compare February 11, 2025 02:35

Oops

201d5e6

thiagoelg approved these changes Feb 11, 2025

View reviewed changes

ljmotta removed the pr: wip PR is still under development label Feb 11, 2025

tiagobento reviewed Feb 11, 2025

View reviewed changes

ljmotta added 3 commits February 12, 2025 07:31

Use build-env

59608d8

Change the image name and update README

1a35f57

Fix streamVersion to streamName

a2645a6

tiagobento approved these changes Feb 12, 2025

View reviewed changes

tiagobento merged commit 23da869 into apache:main Feb 12, 2025
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

ljmotta commented Jan 21, 2025 •

edited

Loading

jomarko commented Jan 22, 2025

ljmotta commented Jan 23, 2025

thiagoelg left a comment

ljmotta commented Jan 28, 2025

tiagobento commented Jan 29, 2025

ljmotta commented Jan 29, 2025 •

edited

Loading

tiagobento commented Jan 31, 2025

ljmotta commented Jan 31, 2025

thiagoelg left a comment

ljmotta commented Feb 11, 2025

tiagobento Feb 11, 2025

ljmotta Feb 12, 2025

ljmotta Feb 12, 2025

tiagobento Feb 11, 2025

tiagobento Feb 11, 2025

tiagobento Feb 11, 2025 •

edited

Loading

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

Conversation

ljmotta commented Jan 21, 2025 • edited Loading

Description

Known Issues

Trying it out

Updating screenshots

jomarko commented Jan 22, 2025

ljmotta commented Jan 23, 2025

thiagoelg left a comment

Choose a reason for hiding this comment

ljmotta commented Jan 28, 2025

tiagobento commented Jan 29, 2025

ljmotta commented Jan 29, 2025 • edited Loading

tiagobento commented Jan 31, 2025

ljmotta commented Jan 31, 2025

thiagoelg left a comment

Choose a reason for hiding this comment

ljmotta commented Feb 11, 2025

tiagobento Feb 11, 2025

Choose a reason for hiding this comment

ljmotta Feb 12, 2025

Choose a reason for hiding this comment

ljmotta Feb 12, 2025

Choose a reason for hiding this comment

tiagobento Feb 11, 2025

Choose a reason for hiding this comment

tiagobento Feb 11, 2025

Choose a reason for hiding this comment

tiagobento Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

ljmotta commented Jan 21, 2025 •

edited

Loading

ljmotta commented Jan 29, 2025 •

edited

Loading

tiagobento Feb 11, 2025 •

edited

Loading