Adding ability to run multiple subsets of JCK tests at once

[terryzuoty]

For the convenience of executing JCK tests, there is a need to run arbitrary subsets of JCK tests (e.g. api/java_lang and api/java_math/BigDecimal).
Based on the description in openjdk.test.jck/include/test_targets.mk, there is a test.jck.custom target which can help developer to execute arbitrary subset of JCK tests. But it seems this target can only accept one test subset at once, if multiple subsets were provided, it will throw exception.

For example:

1. cmd like this make -k test.jck.custom JCKVERSION=jck9 JCKTESTSUITE=RUNTIME JCKTEST="api/java_lang" will successfully generate the STF JCK test script and start running the api/java_lang JCK test.

2. cmd like this make -k test.jck.custom JCKVERSION=jck9 JCKTESTSUITE=RUNTIME JCKTEST="api/java_lang api/java_math/BigDecimal" will failed the STF JCK test execution with error message

STF 18:06:36.920 - Redirecting stderr to /tmp/stf/20171109-180633-Jck/results/3.JCK.stderr
STF 18:06:36.920 - Redirecting stdout to /tmp/stf/20171109-180633-Jck/results/3.JCK.stdout
STF 18:06:36.933 - Monitoring processes: JCK
JCK stderr + set jck.env.runtime.testExecute.otherOpts "   --add-modules java.xml.ws.annotation,java.xml.bind,java.xml.ws,java.activation,java.corba -Xdump:system:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -Xdump:snap:none -Xdump:snap:events=gpf+abort+traceassert+corruptcache -Xdump:java:none -Xdump:java:events=gpf+abort+traceassert+corruptcache -Xdump:heap:none -Xdump:heap:events=gpf+abort+traceassert+corruptcache "
JCK stderr Configuration file can not be imported due to question with specified tag "jck.env.runtime.testExecute.otherOpts" was not found on path. The path is:jck.intro
JCK stderr jck.env.simpleOrAdvanced (advanced)
JCK stderr jck.env.envName (jck_runtime)
JCK stderr jck.env.description (JCK9 runtime template jti for ST...)
JCK stderr jck.env.platform (jre)
JCK stderr jck.tests.chooseTests (Yes)
JCK stderr jck.tests.treeOrFile (tree)
JCK stderr jck.tests.tests (api/java_lang api/java_math/BigD...)
JCK stderr jck.env.testPlatform.intro
JCK stderr jck.env.testPlatform.nativeCode (Yes)
JCK stderr jck.env.testPlatform.multiJVM (Yes)
JCK stderr jck.env.testPlatform.useAgent (No)
JCK stderr jck.env.testPlatform.os (Current system)
JCK stderr jck.env.runtime.intro
JCK stderr jck.env.runtime.otherJVM
JCK stderr jck.env.runtime.testExecute.intro
JCK stderr jck.env.runtime.testExecute.cmdAsFile (/java9/bin/java)
JCK stderr jck.env.runtime.testExecute.classpath (command line option)
JCK stderr jck.env.runtime.testExecute.classpathOpt (-classpath #)
JCK stderr jck.env.runtime.testExecute.additionalClasspath ()
JCK stderr jck.env.moduleSystem.vmOptions (vmAddModsOptionTemplate=--add-mo...)
JCK stderr jck.env.runtime.testExecute.nativeLibsLinkage (dynamic)
JCK stderr jck.env.runtime.testExecute.libPath (environment variable)
JCK stderr jck.env.runtime.testExecute.libPathEnv (will_be_set_by_test_automation_a...)
JCK stderr jck.env.runtime.testExecute.nativeLibrariesLocation (Yes)
JCK stderr jck.env.runtime.testExecute.nativeLibPathFileValue (will_be_set_by_test_automation_a...)
JCK stderr jck.env.runtime.testExecute.cannot_read
JCK stderr 
STF 18:06:38.599 - **FAILED** Process JCK ended with exit code (3) and not the expected exit code/s (0)
STF 18:06:38.599 - Monitoring Report Summary:
STF 18:06:38.599 -   o Process JCK ended with exit code (3) and not the expected exit code/s (0)
STF 18:06:38.600 - Killing processes: JCK
STF 18:06:38.600 -   o Process JCK is not running
**FAILED** at step 3 (Running JCK in multijvm way with Agent off). Expected return value=0 Actual=1 at /tmp/stf/20171109-180633-Jck/execute.pl line 93.
STF 18:06:38.619 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 18:06:38.620 - 
STF 18:06:38.620 - ====================   T E A R D O W N   ====================

Although we can use a loop (in script) to run a list of subsets, the test summary outputs will be separated and hard to track which tests passed and which tests failed. I think we need the STF JCK test to be able to run multiple subsets of JCK test at once, as we want the final test summary to be combined together.

[terryzuoty]

Per talk with Simon, each JCK test run will generate a .jtb file. If we defined multiple test targets in one run, the .jtb file will be overridden by other test settings, which may result in unexpected test failure due to mismatching test environment settings.

A suggested solution is to create extra makefile targets for subdirectories. So if we want to use this method, I think one question is where these targets should be added to? a) Adding them to the main test_targets.mk file or create a new one and include it from the test_targets.mk?

Also, if subsets of JCK tests are executed from makefile targets, I think there needs a mechanism to organize the whole test summary output. For example, if target test.jck9.runtime.subset includes api/java_lang/instrument api/java_lang/invoke api/java_lang/reflect, the final test summary of make test.jck9.runtime.subset would be more readable if each test target's summary are combined together but not separated into parts. If this is agreed, I think a separate issue is needed to tracking this feature.

Hi @lumpfish, I open this issue and moved our talk here to keep them under tracking. Could you kindly give some suggestions upon the question above?

FYI @smlambert @pshipton

[pshipton]

Having all the separate results combined into one isn't a requirement. I suggest leaving it for a future enhancement. As long as we can easily see which subset failed, we can open the results for that subset to see the details.

Although there are many failures now, in the future all the tests will be passing and should remain passing. We'll only be interested in the rare occurrences when something starts failing as the result of a merged change.

[terryzuoty]

Great, thanks for confirming Pete