General Retrospective for July 2024 Releases

[smlambert]

Summary

A retrospective for all efforts surrounding the titular releases.

All community members are welcome to contribute to the agenda via comments below.

This will be a virtual meeting after the release, with at least a week of notice in the #release Slack channel.

On the day of the meeting we'll review the agenda and add a list of actions at the end.

Invited: Everyone.

Time, Date, and URL

Time: 3pm (GMT+1, UK), 10am (GMT -4, Ontario)
Date: Wednesday, 31st of July
URL: https://meet.google.com/tny-iobv-omt
Phone: +44 20 3937 3489 and enter this PIN: 698 738 573#
To view more phone numbers, click this link: https://tel.meet/tny-iobv-omt?hs=5

Details

Retrospective Owner Tasks (in order):

• Post retro URL in #Release around the start of the new release.
• Wait until most builds are released, with no signs of a respin.
• Announce the retrospective's date + time on #Release a week in advance.
• Host the retrospective:
  • Go through the agenda.
  • Create a list of actions.
• Process each action:
  • Create a "WIP" issue including the source comment.
  • Add the issue to the current iteration.
  • Add an issue link to the action list.
• Create a new retrospective issue for the next release.
• Set a calendar reminder so you remember to do step 1 before the next release.
• Close this issue.

TLDR

Add proposed agenda items as comments below.

[smlambert]

Would be handy to link to the release pipeline generator job in the Checklist (instead of having to go to release doc to find it)

[smlambert]

Update release status template to include aarch64_alpine-linux where applicable

[smlambert]

Update code freeze message from

With under a week to go until releases, we are entering a lockdown period for the master branches in the following repositories: github-release-scripts, containers, installer, and mirror-scripts. The temurin-build, ci-jenkins-pipelines and jenkins-helper master branches are still open for work, however the release branch will be under code-freeze management.

as it is not a 'lockdown period for master branches', but for the release branches.

[andrew-m-leonard]

It is not possible to dryrun jdk22u, as there is no jdk-22.0.2 upstream tags, and I don't feel we should artificially create a arbitrary "tag".
I think we should probably enable for these Oracle managed releases, to be able to launch a release-pipeline "Release" type build using the HEAD "dev" branch rather than a tag...?

[andrew-m-leonard]

It is not possible to dryrun jdk22u, as there is no jdk-22.0.2 upstream tags, and I don't feel we should artificially create a arbitrary "tag". I think we should probably enable for these Oracle managed releases, to be able to launch a release-pipeline "Release" type build using the HEAD "dev" branch rather than a tag...?

Interesting, I’ve just manually kicked off a release-openjdk22-pipeline and it’s actually building HEAD of “dev” with a "version string" of the latest ga tag which is 22.0.1+8-ga
I was expecting the testenv check to fail…but it hasn’t I think because scm_ref is “”
So that will probably do, as it is building the very latest jdk-22.0.2 HEAD commits, but under a version string of jdk-22.0.1+8, which is probably fine as a dryrun

[smlambert]

I did not get adoptium/aqa-tests#5456 merged until after pipelines triggered, some tests may have been triggered without the excludes in place. Reruns should pick up the changes.

[jiekang]

Some test jobs failed as the test gathering process was sent to a machine that could not complete the job. We should have worker labels on nodes that are safe for these jobs.

Text below from:
adoptium/aqa-tests#5444 (comment)

x86-64_windows 21

extended.perf

Failed when collecting artifacts via machine test-sxa-armv7l-ubuntu2004-odroid-2

[2024-07-17T06:00:38.100Z] Running on test-sxa-armv7l-ubuntu2004-odroid-2 in /ssd/jenkins/workspace/Test_openjdk21_hs_sanity.perf_x86-64_windows
...
[2024-07-17T06:04:49.333Z] Exception: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException

https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.perf_x86-64_windows/159/consoleText

[smlambert]

Scorecards make assumption that a -ga tag has dropped. There has been no -ga tag for jdk8u422, https://github.com/adoptium/jdk8u/tags, so our calculations / scores won't get generated (and when they do, we will have to adjust based on a mailing list email or when we finally decided to trigger off of jdk8u422-b05

[tellison]

The marketplace data repository had it's update action suspended due to inactivity in the repository.

Presumably this was notified by e-mail to some group (at Eclipse?) - it was only noticed by the project by chance. We need to receive such notifications and/or configure the repo to keep the actions alive longer.

[steelhead31]

Solaris AQA Pipeline failed due to issues with USE_TESTENV_PROPERTIES, the shell script it calls, does not work on solaris 

https://github.com/adoptium/aqa-tests/blob/master/scripts/testenv/testenvSettings.sh

Falls out with the IF statement on line 5

[smlambert]

Update checklist to use correct link for homebrew cask https://github.com/Homebrew/homebrew-cask/blob/master/Casks/t/temurin.rb

[smlambert]

Same issue as previous releases, #28 (comment), where Release Notes do not show up on website even after a day of them being published to the binaries repo, https://adoptium.net/temurin/release-notes/?version=jdk-21.0.4+7

[smlambert]

There are 2 mentions in checklist to check for JDK8 arm32 tags, one refers to Mercurial, is it outdated and if so, should be removed
See the item that reads: "Check for presence of jdk8u aarch32 GA tag and mirror it Mercurial repo - Mirror job"

[smlambert]

Update the link that is listed in this checklist item, since it doesn't exist.

"Consider updating the API as required via the relevant parts of the Adoptium API model constants. "

[andrew-m-leonard]

Unfortunately we missed a jdk8u Mac x64 "sign_installer" failure: https://adoptium.slack.com/archives/CLCFNV2JG/p1721905189045429?thread_ts=1721901821.714339&cid=CLCFNV2JG

This highlights the problem of always RED build pipelines, as test job failures are propagated...?

Propose: We should re-order the pipeline such that signInstaller, gpgSign and verifySigning stages are all done BEFORE running aqa-tests, if any of those 3 stages fail, we do not run aqa-tests. This then makes the failure very obvious to the release AQA triage.
https://github.com/adoptium/ci-jenkins-pipelines/blob/69247d65158df1a49fdea5c797d57122d4b05c61/pipelines/build/common/openjdk_build_pipeline.groovy#L2215

Issue & PR to re-order: adoptium/ci-jenkins-pipelines#1087

[smlambert]

We should re-order the pipeline such that signInstaller, gpgSign and verifySigning stages

+1 as this would also save resources to not trigger testing at all in the cases where those stages fail

[andrew-m-leonard]

The generated dry-run Publish job link for "Point release" re-builds, does not add the adopt_build_number to the TAG parameter

[smlambert]

Consider just adding all of the upcoming releases to the release calendar in 1 go and removing this item from the release checklist

Ensure the adoptium calendar is updated for the next cycle at a minimum

[andrew-m-leonard]

Mac homebrew cask updates process is not clear, especially for ".1" re-build updates, eg.Homebrew/homebrew-cask#180691
Documentation not great if you've not done it before........?

[smlambert]

Notes from aqa-tests triage:

JDK8

• x64 Solaris: 2 SSL testcase failures, javax/net/ssl/ServerName/SSLEngineExplorerMatchedSNI.java & sun/management/jmxremote/bootstrap/SSLConfigFilePermissionTest.sh suspect due to new machine config, consider non-blocking
• arm32 linux and win32: Exclusion needed for compiler/loopopts/TestUnrollLimitPreciseType.java#test1 on this platform, using Unrecognized VM option 'MaxVectorSize=64'
• Decide what to do with TestTLS12 that fails on Ubuntu 24.04 nodes JDK8 fips/TestTLS12 failure on Ubuntu 24.04 aqa-tests#5420

JDK17

• CheckCICompilerCount and GetXSpace fail and need investigation exclusion, non-blocking on arm_linux

JDK22

• x64 Windows: com/sun/net/httpserver/simpleserver/CommandLinePositiveTest.java and simpleserver/jwebserver/CommandLinePositiveTest.java are problematic for JDK22 and JDK21, address in use failures
• aarch64 Linux, 2 MBCS targets need fixing or exclusion, non-blocking
• riscv64: Decide if we should exclude SuspendResume1 since it can only pass on certain nodes

[smlambert]

The new reduced size rerun feature in aqa-tests is great, really reduced time it took to triage tests.

[smlambert]

Was great that @steelhead31 did all the installer items in the checklist. Wondering if there will be updates to that process before October.

[sophia-guo]

Should we add the publish aqa-tests result taps tar to the checklist? Also somewhere to add the documents of how to do this?

[adamfarley]

Actions

Adam

• These simple changes in a single PR:
  • General Retrospective for July 2024 Releases #50 (comment) - Already present, see "Generate release pipeline jobs"
  • General Retrospective for July 2024 Releases #50 (comment)
  • General Retrospective for July 2024 Releases #50 (comment)
  • General Retrospective for July 2024 Releases #50 (comment)
  • Remove this line: General Retrospective for July 2024 Releases #50 (comment)
  • Done. Link
• Link issue already raised for: General Retrospective for July 2024 Releases #50 (comment)
  • May be this issue.
• Raise eclipse issue and add action to release checklist: General Retrospective for July 2024 Releases #50 (comment) Issue raised Checklist change PR.
• Raise issue to make sure that these are sent to a particular label (whatever the label ends up being) General Retrospective for July 2024 Releases #50 (comment) Issue raised.
• Raise and assign issue to Stewart for sorting out access to release calendar: General Retrospective for July 2024 Releases #50 (comment) Issue raised.
• Raise infra issue for the Solaris line: General Retrospective for July 2024 Releases #50 (comment) Issue raised.
• Raise AQA issue for CheckCICompilerCount and GetXSpace: General Retrospective for July 2024 Releases #50 (comment) Issue raised.
• x64 windows: raise issue, assign to scott: General Retrospective for July 2024 Releases #50 (comment) Issue raised.

Andrew

• General Retrospective for July 2024 Releases #50 (comment)
• Document this process (also, see the comment below it too): General Retrospective for July 2024 Releases #50 (comment)
• Raise issue: General Retrospective for July 2024 Releases #50 (comment) : dry-run refactor publish generated job and link does not handle adopt_build_number ci-jenkins-pipelines#1088
• Add clarification (maybe just "has it already been done before") : General Retrospective for July 2024 Releases #50 (comment)

Shelley

• Create short-form release checklist for personal use (and possible later contribution).
• Exclude Arm32 tests mentioned here: General Retrospective for July 2024 Releases #50 (comment)
• Raise issue for this: General Retrospective for July 2024 Releases #50 (comment)

Sophia

• General Retrospective for July 2024 Releases #50 (comment)

Other notes
Already fixed: #50 (comment)