EPIC: Automate & Improve Production Of Linux Package Installers

[steelhead31]

There are currently a number of issues when producing the Linux APKs, Debs & RPMs for the JDKs / JREs at the end of a release cycle:

Notably:

• Currently take ~1 week at the end of a release
• Quite painful for CPU when doing 5 major version releases
• Lots of manual updates required to numerous files, which can be prone to error.
• Lots of work to add new versions and O/S distributions
• All architectures for a version have to be completed, before installer packages can be produced
• Removing a single architecture (when required) is even more time consuming and error prone.
• Currently doing platform specific installers is quite complicated

Proposal:

Ideally, the linux packages should be produced on a per version/architecture basis, and this should be done at the same time that the linux distribution tar balls are published to Github.

The new process, might be as follows:

When the release binaries are published to github via the jenkins refactor release tool :  

• Run a subsequent process to create the package installers, for each dist/version/architecture combination.
• Once the linux installer packages are produced, push them to artifactory

This will require the following tasks:

• Create a generic script that can build an individual arch, version, distro combination
• Dynamically generate the APK build, debian control and RPM spec files.
• For the job that generates files, how do we keep track of those files (e.g. pushed into a repo?)
  • Ideally tracked in a GitHub repository

Advantages of this approach :

• The process doesn’t disrupt the existing steps to test the installers

Improvement Items That Can Be Added:

• Improve handling of src rpms to be architecture specific, rather than a single huge file, with EVERY architecture contained.
• For the past issue where pushing to JFrog has failed
  • Need a follow-on verification test for whether an artifact was pushed successfully on JFrog
  • Download the artifact from jfrog and perform a simple test.
• Also looking at using symlinks to reduce the number of uploads to JFrog, though the technical work is ongoing
• Add metrics for how long it takes artifacts to be pushed to JFrog, but when we do improve this, we will also look back at previous releases to see how big the improvement is
  • Would also be nice to get metrics for the container images
• AIX installers don’t exist at this time

List Of Tasks:

• Linux Packages : Produce Outline Jenkins Pipeline POC process #1019
• Linux Package Rework - Write Process To Validate Parameters From Triggering Job & Validate Artifacts #1037
• Add JDK8 version handling support ( Part of 1037 )
• Add JDK8 ARM32 version handling support ( Part of 1037 )
• Amend jenkinsfile to have stage specific nodes.
• Investigate refactoring of Linux installer gradle build process to aid automation #1049
• Create all distribution templates for installer automation process #1069
• Create Python Script To Create Package Build Files From APKBuild / deb / specfile Templates #1045
• Installer Refactor - Initial Test Of Current Platforms/Versions/Architectures #1072
• Installer Refactor : Refactor package uploads for automation #1080
• How to check/track what has been completed as part of a release / ensure nothing is missed.
• Allow process to work with a multiple publish, rather than individual tarballs.
• Create Jenkins equivalent of current process to produce linux packages, to be used outside of the release automation

FUTURE ENHANCEMENTS:

• Enhance script error reporting for "expected" issues, ie no source ball on github etc.
• Add ability to automatically calculate package release number
• Checkin files created by above process - DISCUSS
• Add process for debian packages to add a new metadata to support a new version
• Allow package builds for EA/nightly without artifactory publish
• Rework Github Actions To Check New Process

[steelhead31]

@gdams feel free to expand the above :)

[jiekang]

Another area to investigate is the need for distribution and version specific repositories for RPMs. As far as I'm aware, we are not building distribution-specific nor version specific artifacts and so we should also be able to publish them to a distribution and version agnostic repository.

For example, the Eclipse Temurin JDK 23 RPM for AmazonLinux 2, Fedora 40 and Fedora 39 are the same file. And other products that provide yum repositories are agnostic:

Amazon Corretto:
baseurl=https://yum.corretto.aws/$basearch

Microsoft VS Code:
baseurl=https://packages.microsoft.com/yumrepos/vscode

However Eclipse Temurin is not:
baseurl=https://packages.adoptium.net/artifactory/rpm/${DISTRIBUTION_NAME:-$(. /etc/os-release; echo $ID)}/\$releasever/\$basearch

We can simplify the Eclipse Temurin structure to something like: https://packages.adoptium.net/artifactory/rpm/$basearch and have significantly less uploads of the same RPM to JFrog.

This is also helpful as we do not need to upload another duplicate artifact every time there is a new version of say Fedora (every 6 months) where users are left broken and waiting for our update when using the current baseurl.

[steelhead31]

Currently prototyping a jenkins process based on the following logical flow..

1. The refactor open-jdk release job is triggered as part of the release pipeline (in dry run mode) ready to be rerun when a platform is ready to be pushed to the github binary repository.

2.  When the above process completes successfully, trigger the new "modular linux package build pipeline", passing through the current platform/arch specific parameters

3. Modular package builder checks if its a release pipeline, ( ie Release is TRUE ), if not exits and does nothing
   Prototype job is here: https://ci.adoptium.net/job/sfr-build-linux-package-modular/

4. If it is a release pipeline, AND dry run is FALSE ( ie binaries are being pushed to github ), then build the linux installer packages.

..... WILL Expand This As I define the process more thoroughly .....

[andrew-m-leonard]

Issues from October release: #1042

[steelhead31]

Future Enhancement: Do dry runs of package builds for EA / nightly builds