jdk_time target fails on some arm32_linux nodes

[smlambert]

Please set the title to indicate the test name and machine name where known.

As indicated by Jan 2024 CPU triage, in adoptium/aqa-tests#4982 (comment), the jdk_time test target was failing with timeouts on ci.adoptium.net arm32 nodes. It was able to pass on the temurin-compliance arm32 linux node.

To make it easy for the infrastructure team to repeat and diagnose, please
answer the following questions:

• test suite/name (e.g, BUILD_LIST, TARGET, CUSTOM_TARGET)? BUILD_LIST=openjdk, TARGET=jdk_time
• a link into recent Test_ job on https://ci.adoptium.net which showed the failure
  • https://ci.adoptium.net/job/Test_openjdk11_hs_extended.openjdk_arm_linux/127/
  • https://ci.adoptium.net/job/Grinder/8603
• Hyperlink to re-run in Grinder:
  • rerun link
• Is there an existing issue elsewhere covering this? No
• Which machine(s) does it work on? jck-equinix-containerized-ubuntu2004-aarch32-1
• Which machine(s) does it fail on? https://ci.adoptium.net/label/hw.arch.aarch32&&ci.role.test&&sw.os.linux/

Any other details:
Suspect related to the findings in #2002 but not confirmed

[sxa]

Will be part of the top level #2662

[Haroon-Khel]

Some jdk_time_2 tests failed on test-docker-ubuntu2004-armv7l-4 with the last jdk17 release
https://ci.adoptium.net/job/Test_openjdk17_hs_extended.openjdk_arm_linux/137/tapResults

Failed test cases: 
TEST: java/time/tck/java/time/format/TCKLocalizedOffsetIdPrinterParser.java
TEST: java/time/tck/java/time/serial/TCKZonedDateTimeSerialization.java
TEST: java/time/tck/java/time/format/TCKLocalizedPrinterParser.java
TEST: java/time/tck/java/time/serial/TCKOffsetDateTimeSerialization.java
TEST: java/time/tck/java/time/temporal/TCKChronoUnit.java
TEST: java/time/tck/java/time/serial/TCKLocalTimeSerialization.java
TEST: java/time/tck/java/time/serial/TCKZoneIdSerialization.java
TEST: java/time/tck/java/time/format/TCKInstantPrinterParser.java
TEST: java/time/tck/java/time/serial/TCKLocalDateSerialization.java
TEST: java/time/tck/java/time/serial/TCKYearSerialization.java
TEST: java/time/tck/java/time/format/TCKSignStyle.java
TEST: java/time/tck/java/time/format/TCKFormatStyle.java
TEST: java/time/tck/java/time/temporal/TCKChronoField.java
TEST: java/time/tck/java/time/format/TCKZoneIdPrinterParser.java
TEST: java/time/tck/java/time/format/TCKPadPrinterParser.java
TEST: java/time/tck/java/time/format/TCKTextStyle.java
TEST: java/time/tck/java/time/serial/TCKPeriodSerialization.java
TEST: java/time/tck/java/time/serial/TCKLocalDateTimeSerialization.java
TEST: java/time/nontestng/java/time/chrono/HijrahConfigTest.java
TEST: java/time/tck/java/time/format/TCKDateTimeParseResolver.java
TEST: java/time/tck/java/time/zone/serial/TCKZoneOffsetTransitionRuleSerialization.java
Test results: passed: 163; error: 23

However all of these tests passed when run individually on the machine itself

jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/format/TCKLocalizedPrinterParser.java    
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/serial/TCKOffsetDateTimeSerialization.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/temporal/TCKChronoUnit.java               
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/serial/TCKLocalTimeSerialization.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/serial/TCKZoneIdSerialization.java   
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/format/TCKInstantPrinterParser.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/serial/TCKLocalDateSerialization.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/serial/TCKYearSerialization.java     
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/format/TCKSignStyle.java        
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/format/TCKFormatStyle.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/temporal/TCKChronoField.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/format/TCKZoneIdPrinterParser.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/format/TCKPadPrinterParser.java   
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/format/TCKTextStyle.java       
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/serial/TCKPeriodSerialization.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/serial/TCKLocalDateTimeSerialization.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/nontestng/java/time/chrono/HijrahConfigTest.java       
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/format/TCKDateTimeParseResolver.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork
jenkins@1a14546b3875:~/test$ $JAVA -jar jtreg/lib/jtreg.jar jdk17u/test/jdk/java/time/tck/java/time/zone/serial/TCKZoneOffsetTransitionRuleSerialization.java
Test results: passed: 1
Report written to /home/jenkins/test/JTreport/html/report.html
Results written to /home/jenkins/test/JTwork

using the latest jdk17 release

jenkins@1a14546b3875:~/test$ $JAVA -version 
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
OpenJDK Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)

[Haroon-Khel]

Looks like it times out when you run them in jdk_time together, but pass when running them individually.

https://ci.adoptium.net/job/Grinder/8698/console

16:17:12  
16:17:12  TEST RESULT: Error. Agent error: java.lang.Exception: Agent 52 timed out with a timeout of 960 seconds; check console log for any additional details
16:17:12  --------------------------------------------------
16:17:45  TEST: java/time/tck/java/time/serial/TCKZonedDateTimeSerialization.java
16:17:45  TEST JDK: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image
16:17:45  
16:17:45  ACTION: build -- Error. Agent error: java.lang.Exception: Agent 53 timed out with a timeout of 960 seconds; check console log for any additional details
16:17:45  REASON: Named class compiled on demand
16:17:45  TIME:   2073.694 seconds

[Haroon-Khel]

Running jdk_time_2 from my own branch, with JTREG_TIMEOUT_OPTION increased to 40 (I dont know how to do it from the grinder)
https://ci.adoptium.net/job/Grinder/8700/console

[Haroon-Khel]

htop on the dockerhost machine while the above grinder is running

grinder 8700 has lost contact with test-docker-ubuntu2004-armv7l-4

17:27:15  XML output with verification to /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17067220302633/jdk_time_2/work
17:47:17  Cannot contact test-docker-ubuntu2004-armv7l-4: java.lang.InterruptedException

[Haroon-Khel]

I think the jdk_time test target might be a bit much for the docker static containers.

The grinder has caused test-docker-ubuntu2004-armv7l-4 to disconnect from jenkins
https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2004%2Darmv7l%2D4/log

SSH Connection failed with IOException: "The kexTimeout (65000 ms) expired.", retrying in 15 seconds. There are 1 more retries left.
The kexTimeout (65000 ms) expired.
ERROR: Connection is not established!
java.lang.IllegalStateException: Connection is not established!
	at com.trilead.ssh2.Connection.getRemainingAuthMethods(Connection.java:989)

[Haroon-Khel]

Rerunning on a non docker container (odroid machine) https://ci.adoptium.net/job/Grinder/8702/console

[Haroon-Khel]

Passed on the odroid machine. Also passed on test-docker-ubuntu2004-armv7l-2 https://ci.adoptium.net/job/Grinder/8701/console

[sxa]

Interesting ...

@Haroon-Khel Are they all new containers and did you cap the CPU/memory on them? From the Grinder logs the run on both containers as mentioned in #3360 (comment) and #3360 (comment) were both running with -concurrency:81 on the jtreg command line (CPU cores/2+1. Those machines have 160 physical cores) but most of the containers should be started with options that restrict the containers to less than that. I'm not quite sure why the two container Grinders didn't show the same behaviour, but the odroid system had concurrency:1 so were potentially safer. I'm slightly surprised that a time related test would give a large load, but maybe we need to restrict those tests. I would have expected similar results on the aarch64 systems though, which are run on the same host 🤷🏻

Noting that the calculation for currency is the one I was looking at adjusting in adoptium/aqa-tests#4792 (in case you want to see where it is and play with it) but I don't think that will affect this particular situation.