Refactor signing/assemble code in openjdk_build_pipeline.groovy to support Windows/docker builds #1117

sxa · 2024-09-26T14:10:32Z

There's a lot in here, so apologies in advance to the reviewers who will need a bit of time to review this properly ;-) Noting that there are some SXAEC: references in here which are things that still need to be cleaned up but the overall architecture and flow is now functional as per the builds in https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/windbld which are now getting through all the steps and running tests.

Some things done in this PR:

A new batOrSh function that runs commands under the windows batch scripts if we're on windows instead of always using bash as this was causing some instability (issue 3714 listed below)
Split out some parts of buildScripts into separate functions: buildScriptsAssemble and buildScriptsEclipseSigner
Each of those two are now listed as a separate "stage" in the pipelines, so we can see how long each part takes. The original sign phase has been renamed as sign tgz/zip to avoid ambiguity. I'm tempted to change assemble to assemble & SBoM (Noting that it also includes the du -k of the output from make-adopt-build-farm.sh which is slow - at least 10 minutes - on my current test machine)
There are a few openjdk_build_pipeline: eyecatchers added so we can more easily find the separate sections while looking at the build logs (curl -s the consoleText link and grep it for _pipeline: - similar to the build.sh: lines from the temurin-build repo)
The setting up of envVars has been extracted into the top level build() and passed into the buildScripts() and buildScriptsAssemble() functions that use it
There is an 'ls -l' of various directories which forces shortname creation. It's a bit of a hack but the shortnames in containers is a bit hit or miss and this makes it reliable ... Without it it won't find various bits of the compiler tools
I've left openjdk_build_dir_arg (obsoleted in Remove use of --user-openjdk-build-root-directory now JDK-8326685 backported #1084) commented out but I'm still tempted to just remove it now ...
Initial stash of files for the internal signing phase now restricts the number of files copied across as **/* was unnecessary.

There are still some issues with the workspace cleaning options which may need to be addressed, although that can be done in subsequent PRs. e.g. rm -rf c:/workspace/openjdk-build/cyclonedx-lib c:/workspace/openjdk-build/security and rm -rf ' + context.WORKSPACE + '/workspace/target/*. Some additional work will be needed before the clean options will work. Ref: adoptium/infrastructure#3723

Fixes adoptium/infrastructure#3709
Supercedes #1103 (as this includes those changes)
Fixes adoptium/infrastructure#3714
Note: There is some potential follow-on work that could be done to tidy up this groovy script overall in #1116

Existing windows pipeline

New pipeline:

Signed-off-by: Stewart X Addison <[email protected]>

github-actions · 2024-09-26T14:10:47Z

Thank you for creating a pull request!

Please check out the information below if you have not made a pull request here before (or if you need a reminder how things work).

Code Quality and Contributing Guidelines

If you have not done so already, please familiarise yourself with our Contributing Guidelines and Code Of Conduct, even if you have contributed before.

Tests

Github actions will run a set of jobs against your PR that will lint and unit test your changes. Keep an eye out for the results from these on the latest commit you submitted. For more information, please see our testing documentation.

In order to run the advanced pipeline tests (executing a set of mock pipelines), it requires an admin to post run tests on this PR.
If you are not an admin, please ask for one's attention in #infrastructure on Slack or ping one here.
To run full set of tests, use "run tests"; a subset of tests on specific jdk version, use "run tests quick 11,21"

sxa · 2024-09-26T14:19:48Z

run tests

(Running at https://ci.adoptium.net/job/build-scripts-pr-tester/job/openjdk-build-pr-tester/1920/)

eclipse-temurin-bot · 2024-09-26T15:07:37Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

sxa · 2024-09-26T15:14:43Z

run tests

(Triggered https://ci.adoptium.net/job/build-scripts-pr-tester/job/openjdk-build-pr-tester/1922/ with refactored cleanWS post build)

eclipse-temurin-bot · 2024-09-26T15:57:48Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

sxa · 2024-09-26T16:04:27Z

run tests

(This one is not using the cached download so has a chance of being a valid test :-) https://ci.adoptium.net/job/build-scripts-pr-tester/job/openjdk-build-pr-tester/1923/ )

eclipse-temurin-bot · 2024-09-26T16:47:41Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

sxa · 2024-09-26T16:49:56Z

run tests

(enableSigner was deleted from some invocations of buildScripts - linter didn't seem to pick it up though 🤔 New run: https://ci.adoptium.net/job/build-scripts-pr-tester/job/openjdk-build-pr-tester/1924 )

eclipse-temurin-bot · 2024-09-26T17:32:18Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

sxa · 2024-09-26T17:48:19Z

run tests

(Running at https://ci.adoptium.net/job/build-scripts-pr-tester/job/openjdk-build-pr-tester/1926/ since some re-runs using the mechanism described in #1057 (comment) first to verify that it should start testing my code once it gets to that point)

eclipse-temurin-bot · 2024-09-27T03:49:12Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

sxa · 2024-09-27T16:24:17Z

run tests

eclipse-temurin-bot · 2024-09-27T21:02:22Z

 PR TESTER RESULT 

✅ All pipelines passed! ✅

Signed-off-by: Stewart X Addison <[email protected]>

sxa · 2024-10-07T18:05:34Z

run tests

Signed-off-by: Stewart X Addison <[email protected]>

eclipse-temurin-bot · 2024-10-08T11:32:44Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

sxa · 2024-10-10T17:01:23Z

Possibly for later PR - reduction in the number of files copied over to the signing job to about 155 in jdk21u from over 0500. I had this in initially and it has been tested as completing ok with the sign_verification job on windows.

// SXAEC - Windows specific 
//                                        includes: "${base_path}/hotspot/variant-server/**/*," +
//                                            "${base_path}/support/modules_cmds/**/*," +
//                                            "${base_path}/support/modules_libs/**/*," +
//                                            // JDK 16 + jpackage needs to be signed as well stash the resources folder containing the
//                                            "${base_path}/jdk/modules/jdk.jpackage/jdk/jpackage/internal/resources/*"

                                        includes: "${base_path}/hotspot/variant-server/**/*.exe," +
                                            "${base_path}/support/modules_cmds/**/*.exe," +
                                            "${base_path}/support/modules_libs/**/*.exe," +
                                            "${base_path}/hotspot/variant-server/**/*.dll," +
                                            "${base_path}/support/modules_cmds/**/*.dll," +
                                            "${base_path}/support/modules_libs/**/*.dll," +
                                            "${base_path}/hotspot/variant-server/**/*.dylib," +
                                            "${base_path}/support/modules_cmds/**/*.dylib," +
                                            "${base_path}/support/modules_libs/**/*.dylib," +
                                            "${base_path}/jdk/modules/jdk.jpackage/jdk/jpackage/internal/resources/*"

EDIT: Re-instated on later version of the PR as it helped to avoid the chmod a+rwX in the non-docker case

Signed-off-by: Stewart X Addison <[email protected]>

sxa · 2024-10-12T10:32:20Z

This version looks ok on windows docker and non-docker cases for versions 11 through 21 and the zip files are producing the same set of files (other than dfferences in the redist dlls) as the main builds ;-)

JDK8 passes with the latest push to "Disable internal sign phase on jdk8u"

Once this is done we'll need to verify reproducibility, but if there are any issues there then they can likely be resolved in a future PR

Signed-off-by: Stewart X Addison <[email protected]>

sxa · 2024-10-12T12:39:02Z

run tests

sxa · 2024-10-12T12:43:05Z

Clean runs using this PR:

Version	Docker	Nondocker
8	#997
11		#991
21	#992	#990

eclipse-temurin-bot · 2024-10-12T16:51:09Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

sxa · 2024-10-14T10:32:37Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎ See the pipeline-build-check below for more information...

Only failure was jdk11u/linux/x64 which failed pulling the build docker image down onto a dynamically provisioned Azure machine:

14:23:02  Running on [build-linux-x64-2d8590](https://ci.adoptium.net/computer/build%2Dlinux%2Dx64%2D2d8590/) in /home/adoptopenjdk/workspace/build-scripts-pr-tester/build-test/jobs/jdk11u/jdk11u-linux-x64-temurin
[14:23:03  + docker pull adoptopenjdk/centos6_build_image
14:26:39  failed to register layer: write /usr/local/gcc/libexec/gcc/x86_64-pc-linux-gnu/7.5.0/cc1plus: no space left on device

Since jdk11u/Linux/aarch64 and Linux/x64 on other versions worked ok this should not be considered a blocker.

sxa · 2024-10-14T14:32:26Z

Note to self on workflow in build()

- context.stage('queue')
  - if (buildConfig.DOCKER_IMAGE) {
    - context.node(label)
      - if (buildConfig.DOCKER_FILE) {
        - context.docker.build(buildScripts(...))
      - else context.ws(worksapce)(context.docker.image(...) { buildScripts(...) (context.ws is windows only)
      - if (signer)
        - buildScriptsEclipseSigner()
        - call context.docker.image(DOCKER_IMAGE).assemble()
  - else // not docker
    -  if windows
      - set workspace, context.ws(workspace) { buildScripts(); buildScritpsEclipseSigner(); buildScriptsAssemble() }
    - else
      - buildScripts(); if (enableSigner) buildScriptsEclipseSigner(); buildScriptsAssemble();
if ( enableSigner ) sign(versionInfo)
if (enableInstallers) { buildInstaller(); if (enableSigner) signInstaller() }
if (enableTests) {...}

sxa · 2024-10-14T16:33:21Z

FYI @andrew-m-leonard I'm not able to verify reproducibility between the 21.0.4+7-ga and a build in docker (Note: The docker builds are using build tools instead of the full MSVC we used for the GA) It gave me 129 differences, 99.61% reproducibility. I haven't dug into what might be different in the files yet however the files I'm getting out of the build seem noticably larger in the windbld#999 build (e.g. jimage.dll is 43736 bytes instead of 30440, jfr.exe is 34008 bytes instead of 20712). This is the case whether the build was in docker or not.

Build	ZIP	SBOM
21.0.4+7-GA	zip	SBOM
windbld#999 (dockerBuild)	zip	SBOM
windbld#1008 (not docker)	zip	SBOM

The cygwin version is different (3.4.6 vs 3.5.4). The compiler version is identical (19.37.32822) There was no --user-openjdk-build-root-directory C:/workspace/openjdk-build/workspace/build/openjdkbuild parameter in mine which there was on the real GA build. BUILD_FULL_NAME in the SBOM is cygwin_nt-10.0-20348-x86_64--server-release for the dockerbuild, but was C:/workspace/openjdk-build/workspace/build/openjdkbuild in the GA

Signed-off-by: Stewart X Addison <[email protected]>

sxa · 2024-10-14T18:51:08Z

run tests

eclipse-temurin-bot · 2024-10-14T22:24:54Z

 PR TESTER RESULT 

✅ All pipelines passed! ✅

sxa · 2024-10-15T09:36:34Z

Some timings for my own reference before this summary table disappears:

andrew-m-leonard · 2024-10-15T10:41:08Z

@sxa Ah of course, the July release still had the --user-openjdk-build-root-directory C:/workspace/openjdk-build/workspace/build/openjdkbuild
This will cause a difference, as the upstream fix for that was only in jdk-21.0.5.
Can you try building with the exact same path?

sxa · 2024-10-15T11:02:57Z

There are still some issues relating to the clean workspace options in the windows/docker case but otherwise this PR is doing what it needs to and we should be able to get it in after the release (for safety reasons)
FYI @andrew-m-leonard and @AdamBrousseau (It would be good to confirm that this won't break OpenJ9's CI)

sxa · 2024-10-17T16:42:17Z

@sxa Ah of course, the July release still had the --user-openjdk-build-root-directory C:/workspace/openjdk-build/workspace/build/openjdkbuild This will cause a difference, as the upstream fix for that was only in jdk-21.0.5. Can you try building with the exact same path?

Testing at https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/windbld/1048/ (outside docker) and https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/windbld/1049/ (in a container). EDIT: That doesn't work as the alternate build_path logic was taken out in https://github.com/adoptium/ci-jenkins-pipelines/pull/1084/files

In reference to our call earlier, it's worth noting that:

https://ci.adoptium.net/job/Grinder/11190 with jdk21/nightly passed the reproducible test. The nightly build is currently 21.0.5+9 and that is what that job used:

14:39:09  =JAVA VERSION OUTPUT BEGIN=
14:39:09  openjdk version "21.0.5-beta" 2024-10-15
14:39:09  OpenJDK Runtime Environment Temurin-21.0.5+9-202410031113 (build 21.0.5-beta+9-ea)
14:39:09  OpenJDK 64-Bit Server VM Temurin-21.0.5+9-202410031113 (build 21.0.5-beta+9-ea, mixed mode, sharing)
14:39:09  =JAVA VERSION OUTPUT END=

A regrind of another run of dev.system (G#11189) with jdk23 on a specific upstream build (version below) fails due to a mismatch in UCRT DLL on test-azure-2 (A change introduced in Add support for Windows Redist DLL as Adoptium DevKit temurin-build#3981). I have initiated G#11194 on a build machine to compare (Noting that the docker images do not currently have the updated MSVS DLLs as they are not yet in my docker images). EDIT: That failed as it still didn't locate the correct MSVS DLLs

14:29:46  =JAVA VERSION OUTPUT BEGIN=
14:29:47  openjdk version "23.0.1-beta" 2024-10-15
14:29:47  OpenJDK Runtime Environment Temurin-23+37-202410121214 (build 23.0.1-beta+37-202410121214)
14:29:47  OpenJDK 64-Bit Server VM Temurin-23+37-202410121214 (build 23.0.1-beta+37-202410121214, mixed mode, sharing)
14:29:47  =JAVA VERSION OUTPUT END=

I am also running https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/windbld/1052/ which is a build of jdk21 head which is NOT marked as a release but includes the REPRODUCIBLE_COMPARE parameter so hopefully that will trigger the "old" comparison job.

Also queued up another test of 21.0.5 with the devkit in place: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/windbld/1054/ which should eliminate all known differences. (Non-release so probably wont' compare - https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/windbld/1056/ is with RELEASE: true

sxa added 3 commits September 26, 2024 14:49

First pass of batOrSh and fixes for windows containers

718ab4a

Signed-off-by: Stewart X Addison <[email protected]>

Refactoring signing code to allow operation in Windows containers

ced2eb6

Signed-off-by: Stewart X Addison <[email protected]>

De-duplicate envVars setup

e83f8f3

Signed-off-by: Stewart X Addison <[email protected]>

github-actions bot added jenkins-pipeline docker testing windows labels Sep 26, 2024

sxa force-pushed the refactor_signing branch from 0f71881 to 457077d Compare September 26, 2024 15:08

sxa force-pushed the refactor_signing branch from 457077d to ebbc774 Compare September 26, 2024 15:40

sxa force-pushed the refactor_signing branch from ebbc774 to c1a357b Compare September 26, 2024 16:02

sxa force-pushed the refactor_signing branch from c1a357b to 4286580 Compare September 26, 2024 16:49

sxa mentioned this pull request Sep 26, 2024

pr-tester improvements to provide more reliable and better coverage #1057

Open

github-actions bot added arm mac labels Sep 26, 2024

sxa force-pushed the refactor_signing branch from 4286580 to c7d1ce7 Compare September 26, 2024 17:34

sxa force-pushed the refactor_signing branch 2 times, most recently from 9d749be to 6c889b5 Compare September 27, 2024 16:23

sxa force-pushed the refactor_signing branch from 6c889b5 to 81504fc Compare September 28, 2024 10:30

Updates following PR creation (inc.linter fixes)

7ede372

Signed-off-by: Stewart X Addison <[email protected]>

sxa force-pushed the refactor_signing branch from 81504fc to 7ede372 Compare October 1, 2024 14:53

sxa mentioned this pull request Oct 3, 2024

WIP: Trial using context.bat instead of context.sh on Windows in docker #1103

Closed

Move post-build workspace clean code into build()

8bfd0eb

Signed-off-by: Stewart X Addison <[email protected]>

chmod -R at start of assemble phase even on non-docker

8501034

Signed-off-by: Stewart X Addison <[email protected]>

sxa force-pushed the refactor_signing branch from 2114a89 to 8501034 Compare October 7, 2024 23:20

sxa added 2 commits October 11, 2024 00:20

Separate clean workspace function

f4e056b

Signed-off-by: Stewart X Addison <[email protected]>

Reinstate windows release definition for jdk11u

fe4efb5

Signed-off-by: Stewart X Addison <[email protected]>

Reduce stashing and avoid pre-unstash chmod in non-docker case

7c5abfd

Signed-off-by: Stewart X Addison <[email protected]>

sxa force-pushed the refactor_signing branch from 943e1e1 to 7c5abfd Compare October 12, 2024 10:38

Disable internal sign phase on jdk8u

4f6709b

Signed-off-by: Stewart X Addison <[email protected]>

Final cleanups of comments and debug

ccc5e28

Signed-off-by: Stewart X Addison <[email protected]>

sxa force-pushed the refactor_signing branch from 4d6d2f2 to ccc5e28 Compare October 14, 2024 18:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor signing/assemble code in openjdk_build_pipeline.groovy to support Windows/docker builds #1117

Refactor signing/assemble code in openjdk_build_pipeline.groovy to support Windows/docker builds #1117

sxa commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024

sxa commented Sep 26, 2024 •

edited

Loading

eclipse-temurin-bot commented Sep 26, 2024

sxa commented Sep 26, 2024 •

edited

Loading

eclipse-temurin-bot commented Sep 26, 2024

sxa commented Sep 26, 2024 •

edited

Loading

eclipse-temurin-bot commented Sep 26, 2024

sxa commented Sep 26, 2024 •

edited

Loading

eclipse-temurin-bot commented Sep 26, 2024

sxa commented Sep 26, 2024 •

edited

Loading

eclipse-temurin-bot commented Sep 27, 2024

sxa commented Sep 27, 2024

eclipse-temurin-bot commented Sep 27, 2024

sxa commented Oct 7, 2024

eclipse-temurin-bot commented Oct 8, 2024

sxa commented Oct 10, 2024 •

edited

Loading

sxa commented Oct 12, 2024 •

edited

Loading

sxa commented Oct 12, 2024

sxa commented Oct 12, 2024 •

edited

Loading

eclipse-temurin-bot commented Oct 12, 2024

sxa commented Oct 14, 2024

 PR TESTER RESULT 

sxa commented Oct 14, 2024

sxa commented Oct 14, 2024 •

edited

Loading

sxa commented Oct 14, 2024

eclipse-temurin-bot commented Oct 14, 2024

sxa commented Oct 15, 2024

andrew-m-leonard commented Oct 15, 2024

sxa commented Oct 15, 2024 •

edited

Loading

sxa commented Oct 17, 2024 •

edited

Loading

Refactor signing/assemble code in openjdk_build_pipeline.groovy to support Windows/docker builds #1117

Are you sure you want to change the base?

Refactor signing/assemble code in openjdk_build_pipeline.groovy to support Windows/docker builds #1117

Conversation

sxa commented Sep 26, 2024 • edited Loading

Existing windows pipeline

New pipeline:

github-actions bot commented Sep 26, 2024

Thank you for creating a pull request!

Code Quality and Contributing Guidelines

Tests

sxa commented Sep 26, 2024 • edited Loading

eclipse-temurin-bot commented Sep 26, 2024

 PR TESTER RESULT 

sxa commented Sep 26, 2024 • edited Loading

eclipse-temurin-bot commented Sep 26, 2024

 PR TESTER RESULT 

sxa commented Sep 26, 2024 • edited Loading

eclipse-temurin-bot commented Sep 26, 2024

 PR TESTER RESULT 

sxa commented Sep 26, 2024 • edited Loading

eclipse-temurin-bot commented Sep 26, 2024

 PR TESTER RESULT 

sxa commented Sep 26, 2024 • edited Loading

eclipse-temurin-bot commented Sep 27, 2024

 PR TESTER RESULT 

sxa commented Sep 27, 2024

eclipse-temurin-bot commented Sep 27, 2024

 PR TESTER RESULT 

sxa commented Oct 7, 2024

eclipse-temurin-bot commented Oct 8, 2024

 PR TESTER RESULT 

sxa commented Oct 10, 2024 • edited Loading

sxa commented Oct 12, 2024 • edited Loading

sxa commented Oct 12, 2024

sxa commented Oct 12, 2024 • edited Loading

Clean runs using this PR:

eclipse-temurin-bot commented Oct 12, 2024

 PR TESTER RESULT 

sxa commented Oct 14, 2024

 PR TESTER RESULT 

sxa commented Oct 14, 2024

sxa commented Oct 14, 2024 • edited Loading

sxa commented Oct 14, 2024

eclipse-temurin-bot commented Oct 14, 2024

 PR TESTER RESULT 

sxa commented Oct 15, 2024

andrew-m-leonard commented Oct 15, 2024

sxa commented Oct 15, 2024 • edited Loading

sxa commented Oct 17, 2024 • edited Loading

sxa commented Sep 26, 2024 •

edited

Loading

sxa commented Sep 26, 2024 •

edited

Loading

sxa commented Sep 26, 2024 •

edited

Loading

sxa commented Sep 26, 2024 •

edited

Loading

sxa commented Sep 26, 2024 •

edited

Loading

sxa commented Sep 26, 2024 •

edited

Loading

sxa commented Oct 10, 2024 •

edited

Loading

sxa commented Oct 12, 2024 •

edited

Loading

sxa commented Oct 12, 2024 •

edited

Loading

sxa commented Oct 14, 2024 •

edited

Loading

sxa commented Oct 15, 2024 •

edited

Loading

sxa commented Oct 17, 2024 •

edited

Loading