jck do not have enabled rerun

[judovana]

Current jcks are split (https://github.com/adoptium/aqa-tests/tree/master/jck/splitter) into the chunks (eg

aqa-tests/jck/runtime.api/playlist.xml

Line 815 in 06f6219

<testCaseName>jck-runtime-api-javax_activity</testCaseName>

) , which once run, looks (from

aqa-tests/jck/jck.mk

Line 152 in 06f6219

GEN_JTB_GENERIC = $(JAVA_TO_TEST) -Djvm.options=$(Q)$(JVM_OPTIONS)$(Q) -Dother.opts=$(Q)$(OTHER_OPTS)$(Q) -cp $(TEST_ROOT)/jck/jtrunner/bin JavatestUtil testRoot=$(TEST_ROOT) jckRoot=$(JCK_ROOT) jckversion=$(JCK_VERSION) workdir=$(REPORTDIR) configAltPath=$(CONFIG_ALT_PATH) testJava=$(JAVA_TO_TEST) riJava=$(JAVA_TO_TEST) task=cmdfilegen spec=$(SPEC) $(APPLICATION_OPTIONS)

) like

===============================================
Running test jck-runtime-api-javax_activity_0 ...
===============================================
jck-runtime-api-javax_activity_0 Start Time: Fri Sep 27 00:42:07 2024 Epoch Time (ms): 1727397727349
variation: NoOptions
JVM_OPTIONS:  
{ \
echo "";	echo "TEST SETUP:"; \
echo "Nothing to be done for setup."; \
mkdir -p "/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../TKG/output_1727397727920/jck-runtime-api-javax_activity_0"; \
cd "/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../TKG/output_1727397727920/jck-runtime-api-javax_activity_0"; \
echo "";	echo "TESTING:"; \
"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.392.b08-7.fc40.2.x86_64/bin/java" -Djvm.options=" " -Dother.opts="" -cp /home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../jck/jtrunner/bin JavatestUtil testRoot=/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/.. jckRoot=/home/tester/jck-via-aqavit-1727351423/aqa-tck jckversion=jck8d workdir="/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../TKG/output_1727397727920/jck-runtime-api-javax_activity_0" configAltPath=/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../jck/jtrunner/config testJava="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.392.b08-7.fc40.2.x86_64/bin/java" riJava="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.392.b08-7.fc40.2.x86_64/bin/java" task=cmdfilegen spec=linux_x86-64  tests=api/javax_activity testsuite=RUNTIME; \
	"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.392.b08-7.fc40.2.x86_64/bin/java" -jar /home/tester/jck-via-aqavit-1727351423/aqa-tck/JCK-runtime-8d/lib/javatest.jar -config /home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../jck/jtrunner/config/jck8d/runtime.jti @"/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../TKG/output_1727397727920/jck-runtime-api-javax_activity_0"/generated.jtb; \
	if [ $? -eq 0 ]; then echo "-----------------------------------"; echo "jck-runtime-api-javax_activity_0""_PASSED"; echo "-----------------------------------"; cd /home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/..;  else echo "-----------------------------------"; echo "jck-runtime-api-javax_activity_0""_FAILED"; echo "-----------------------------------"; fi; \
	"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.392.b08-7.fc40.2.x86_64/bin/java" -Djvm.options=" " -Dother.opts="" -cp /home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../jck/jtrunner/bin JavatestUtil testRoot=/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/.. jckRoot=/home/tester/jck-via-aqavit-1727351423/aqa-tck jckversion=jck8d configAltPath=/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../jck/jtrunner/config workdir="/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../TKG/output_1727397727920/jck-runtime-api-javax_activity_0" spec=linux_x86-64 task=summarygen tests=api/javax_activity testsuite=RUNTIME; \
echo "";	echo "TEST TEARDOWN:"; \
echo "Nothing to be done for teardown."; \
 } 2>&1 | tee -a "/home/tester/jck-via-aqavit-1727351423/aqa-tck-work/aqa-tests/TKG/../TKG/output_1727397727920/TestTargetResult";

That suggests that jck simply runs once. However by combination of existing, correct workdit and running all non passed tests:

  if [[ -d "$JCK_DIR/JCK-$SUIT-$MAJOR-work" ]] ; then
    local -r CREATE=''
  else
    local -r CREATE="-create"
  fi
...
-workdir $CREATE $JCK_DIR/JCK-$SUIT-$MAJOR-work \
...
-verbose:date,non-pass,start,stop

It is easy to achieve very stable rerun mechanism, which will iterate for known number of iterations or until full pass, depending what comes first. If the #5654 will be implemented, then any subsequent run then first, should be on concurency 1. Advantage is, that final result xml should contain only final version of accumulated passes.

wdyt? (note, this is battle tested in RH, we even have various reruns slightly different - first headless, and some tests excluded, second headfull and still high concurency, and then few loops on concrency 1, and then the remaining tests. As current AQA runners are pretty stable, I would like to not get into this depth just simply looping (especially with #5654 in) few times)

[smlambert]

We have a different rerun mechanism that is used for all types of tests. At the end of a test job, the testcase failures are gathered, and just those failures are rerun when RERUN_ITERATIONS=1 (or some value above the default value of 0).

[judovana]

I know that one. And is awesome! Still in case of TCK, even one subset is quite long, and it may be better to reuse the in-tck mechanism to get rid of one or two false negatives right out of thebox.

[smlambert]

Let's chat on Tuesday when I am back from vacation.

[judovana]

Sure! This is just idea, no enforcement. The individual susbets are very Very VERY stable, so including embedded rerun may add unnecessary complexity.