Add shaInfo target to collect all shas after the test run #651

sophia-guo · 2024-12-12T15:45:58Z

add shaInfo target, which be called before the testSummary
update getSha.txt to getShas.txt, which check all folders under TEST_ROOT, if there is .git then check the SHA and save the repo and sha
remove the ant macro checkGitRepoSha

Signed-off-by: Sophia Guo <[email protected]>

sophia-guo · 2024-12-12T15:54:05Z

functional.dev https://ci.adoptium.net/view/Test_grinder/job/Grinder/12068/
sanity.perf https://ci.adoptium.net/view/Test_grinder/job/Grinder/12069/

smlambert · 2024-12-12T18:11:24Z

This is a nice way to handle test material that is pulled in via git clone. I like it.

Separately we will need to think about how to deal with test material that is precompiled and run from jar files (such as dacapo and renaissance).

We need to check with others who use AQAvit before removing the ant macro, in case it is in use by anyone for internal vendor test material.

sophia-guo · 2024-12-13T15:25:23Z

Have to work with adoptium/aqa-tests#5819

llxia · 2024-12-13T20:42:42Z

settings.mk

@@ -364,7 +364,7 @@ $(SUBDIRS_TESTTARGET):

 $(TESTTARGET): $(SUBDIRS_TESTTARGET)

-_$(TESTTARGET): setup_$(TESTTARGET) rmResultFile $(TESTTARGET) resultsSummary teardown_$(TESTTARGET)
+_$(TESTTARGET): setup_$(TESTTARGET) rmResultFile $(TESTTARGET) shaInfor resultsSummary teardown_$(TESTTARGET)


Does it make sense to have shaInfor before running the test? In the event of a crash, we will still retain the SHA information.

Could we rename shaInfor to shaInfo?

It can be done right after make compile. https://ci.adoptium.net/view/Test_grinder/job/Grinder/12117/ https://ci.adoptium.net/view/Test_grinder/job/Grinder/12118/

However it depends on what the crash is. If the crash exit the job instantly ( https://ci.adoptium.net/view/Test_grinder/job/Grinder/12124/console, this is not a test crash, still explain the scenario) , SHAs.txt is available but *.tap file unavailable, which won't help jenkins story unless we also archive it.

Updated to shaInfo

llxia · 2024-12-13T20:45:32Z

09:47:07  Collect all REPOs sha 
09:47:07  cd "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/scripts"; \
09:47:07  bash "getSHAs.sh" --test_root_dir "/home/jenkins/workspace/Grinder/aqa-tests/TKG/.." --shas_file "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/SHAs.txt" 
09:47:07  Check shas in /home/jenkins/workspace/Grinder/aqa-tests/TKG/.. and store the info in /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/SHAs.txt
09:47:07  touch /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/SHAs.txt
09:47:07  ================================================
09:47:07  timestamp: 20241212-144705
09:47:07  repo dir: /home/jenkins/workspace/Grinder/aqa-tests/TKG/../functional/security/Crypto/CryptoTest
09:47:07  git repo: 
09:47:07    Fetch URL: https://github.com/rh-openjdk/CryptoTest.git
09:47:07  sha:
09:47:07  f20591eb422ecfc2caef43fe2ce6100271e667b3
09:47:07  ================================================
09:47:07  timestamp: 20241212-144705
09:47:07  repo dir: /home/jenkins/workspace/Grinder/aqa-tests/TKG/../functional/security/ssl-tests/ssl-tests
09:47:07  git repo: 
09:47:07    Fetch URL: https://github.com/rh-openjdk/ssl-tests.git
09:47:07  sha:
09:47:07  79f368f9793ab015061b6f4c6010fb3c749491a5
09:47:07  ================================================
09:47:07  timestamp: 20241212-144705
09:47:07  repo dir: /home/jenkins/workspace/Grinder/aqa-tests/TKG/../functional/testHeadlessComponents/TestHeadlessComponents
09:47:07  git repo: 
09:47:07    Fetch URL: https://github.com/rh-openjdk/TestHeadlessComponents.git
09:47:07  sha:
09:47:07  633fa62bde9357a2a454534004bc8c5bba4034da
09:47:07  ================================================
09:47:07  timestamp: 20241212-144705
09:47:07  repo dir: /home/jenkins/workspace/Grinder/aqa-tests/TKG/..
09:47:07  git repo: 
09:47:07    Fetch URL: https://github.com/sophia-guo/openjdk-tests.git
09:47:07  sha:
09:47:07  5e1b49fd81e1a765a13ef2261201512db3d8a3c5
09:47:07  ================================================
09:47:07  timestamp: 20241212-144705
09:47:07  repo dir: /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG
09:47:07  git repo: 
09:47:07    Fetch URL: https://github.com/sophia-guo/TKG.git
09:47:07  sha:
09:47:07  1d6dbc1f8ae0df928f511de95c57bac5a4f14bda

I do not see openj9 SHA in your Grinder.

sophia-guo · 2024-12-16T13:39:59Z

SHA will be collected only if the REPO is used. That is say for openj9 if you run openj9 tests, the sha will be there.

My grinder is testing functional.dev and sanity.perf, which doesn't include openj9 tests. The reason for choosing those two categories is currently the SHAs for those two categories are missing. Similar to grinder if running openjdk tests against impl=hotspot there will be no openj9 repo SHA.

llxia · 2024-12-16T14:18:05Z

Openj9 sha was not captured even if we ran test in openj9 repo.

09:06:46  get functional test material...
09:06:46  git clone --depth 1 --reference-if-able /home/jenkins/openjdk_cache -b master https://github.com/eclipse/openj9.git
09:06:46  Cloning into 'openj9'...

09:07:35  Collect all REPOs sha 
09:07:35  cd "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/scripts"; \
09:07:35  bash "getSHAs.sh" --test_root_dir "/home/jenkins/workspace/Grinder/aqa-tests/TKG/.." --shas_file "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/SHAs.txt" 
09:07:35  Check shas in /home/jenkins/workspace/Grinder/aqa-tests/TKG/.. and store the info in /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/SHAs.txt
09:07:35  touch /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/SHAs.txt
09:07:35  ================================================
09:07:35  timestamp: 20241216-140734
09:07:35  repo dir: /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG
09:07:35  git repo: 
09:07:35    Fetch URL: https://github.com/sophia-guo/TKG.git
09:07:35  sha:
09:07:35  1b194139d7c6f915905d3984a5dfc16aba9a0618
09:07:35  ================================================
09:07:35  timestamp: 20241216-140734
09:07:35  repo dir: /home/jenkins/workspace/Grinder/aqa-tests/TKG/..
09:07:35  git repo: 
09:07:35    Fetch URL: https://github.com/sophia-guo/openjdk-tests.git
09:07:35  sha:
09:07:35  c402bf749616655dc2d6d4ddcb51c1b7953a8ff6

https://ci.adoptium.net/view/Test_grinder/job/Grinder/12101/console

sophia-guo · 2024-12-16T20:56:14Z

openj9 repo is removed once the material is moved https://github.com/adoptium/aqa-tests/blob/master/get.sh#L664, similar to vendors repo information. Hence no .git and no sha information.

Those information is still available by testenv.properties, which can be combined to SHAs.txt.

Or it's not necessary to remove those repo the workspace will be deleted for jenkins run. https://ci.adoptium.net/view/Test_grinder/job/Grinder/12113/ - dynamic compile
https://ci.adoptium.net/view/Test_grinder/job/Grinder/12114/ - non dynamic compile

sophia-guo · 2024-12-16T21:13:57Z

I'm also thinking if the testenv.properties is necessary with *.tap get all SHAs information though it's a separate issue I will take a look later.

smlambert · 2024-12-17T01:08:02Z

From a 'secure dev' perspective, if we are using anything from the a repo, even if its a makefile or a variable or two are set because of it (https://github.com/eclipse-openj9/openj9/blob/master/test/functional/testVars.mk), we should keep track the repo and the SHAs. What we ultimately want to be able to do is to fully reproduce a test run down to every detail.

sophia-guo · 2024-12-17T16:15:49Z

See https://github.com/adoptium/aqa-tests/pull/5819/files, which won't remove openj9 or vendor's repo. So all SHAs can be collected by shaInfo targets

https://ci.adoptium.net/view/Test_grinder/job/Grinder/12113/ - dynamic compile
https://ci.adoptium.net/view/Test_grinder/job/Grinder/12114/ - non dynamic compile

Alternatively is for openj9 and vendors repos call getSHAs.sh in get.sh as before. I personally would prefer the first solution to keep the process simple and well maintained. Changes for both solutions will be in aqa-tests repo. This PR won't be affected.

llxia · 2024-12-17T17:07:02Z

See https://github.com/adoptium/aqa-tests/pull/5819/files, which won't remove openj9 or vendor's repo.

We work with multiple vendor repos (5 vendor repos and soon to be 6). Leaving unused material in the workspace can create confusion.

sophia-guo · 2024-12-17T21:14:37Z

All vendors repos are cloned to its own destination https://github.com/adoptium/aqa-tests/blob/master/get.sh#L719 I'm not sure why it causes confusion.

I don't mind keep removing them if this is preferred.

PR adoptium/aqa-tests#5819 updated.

https://ci.adoptium.net/view/Test_grinder/job/Grinder/12131/
https://ci.adoptium.net/view/Test_grinder/job/Grinder/12132/

sophia-guo · 2024-12-21T04:44:36Z

PR adoptium/aqa-tests#5819 updated, which will delete openj9 and vendors repo, only keep .git folder, so repourl and sha can still be checked in the same way as all others, using shaInfo.
https://ci.adoptium.net/view/Test_grinder/job/Grinder/12146/
https://ci.adoptium.net/view/Test_grinder/job/Grinder/12145/

result

67a6243

Signed-off-by: Sophia Guo <[email protected]>

sophia-guo marked this pull request as draft December 12, 2024 15:46

This was referenced Dec 12, 2024

Move SHA check function to TKG adoptium/aqa-tests#5819

Draft

Track all test materials or frameworks sha info adoptium/aqa-tests#5818

Open

sophia-guo requested a review from smlambert December 12, 2024 16:14

Records all SHAs

1b19413

sophia-guo force-pushed the results branch from 1d6dbc1 to 1b19413 Compare December 12, 2024 16:18

sophia-guo marked this pull request as ready for review December 12, 2024 16:20

karianna approved these changes Dec 12, 2024

View reviewed changes

sophia-guo requested a review from llxia December 13, 2024 15:25

llxia reviewed Dec 13, 2024

View reviewed changes

update shaInfor to shaInfo

29909e3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add shaInfo target to collect all shas after the test run #651

Add shaInfo target to collect all shas after the test run #651

sophia-guo commented Dec 12, 2024 •

edited

Loading

sophia-guo commented Dec 12, 2024

smlambert commented Dec 12, 2024

sophia-guo commented Dec 13, 2024

llxia Dec 13, 2024 •

edited

Loading

sophia-guo Dec 16, 2024 •

edited

Loading

sophia-guo Dec 17, 2024

llxia commented Dec 13, 2024

sophia-guo commented Dec 16, 2024 •

edited

Loading

llxia commented Dec 16, 2024

sophia-guo commented Dec 16, 2024 •

edited

Loading

sophia-guo commented Dec 16, 2024

smlambert commented Dec 17, 2024

sophia-guo commented Dec 17, 2024

llxia commented Dec 17, 2024

sophia-guo commented Dec 17, 2024 •

edited

Loading

sophia-guo commented Dec 21, 2024

Add shaInfo target to collect all shas after the test run #651

Are you sure you want to change the base?

Add shaInfo target to collect all shas after the test run #651

Conversation

sophia-guo commented Dec 12, 2024 • edited Loading

sophia-guo commented Dec 12, 2024

smlambert commented Dec 12, 2024

sophia-guo commented Dec 13, 2024

llxia Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

sophia-guo Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

sophia-guo Dec 17, 2024

Choose a reason for hiding this comment

llxia commented Dec 13, 2024

sophia-guo commented Dec 16, 2024 • edited Loading

llxia commented Dec 16, 2024

sophia-guo commented Dec 16, 2024 • edited Loading

sophia-guo commented Dec 16, 2024

smlambert commented Dec 17, 2024

sophia-guo commented Dec 17, 2024

llxia commented Dec 17, 2024

sophia-guo commented Dec 17, 2024 • edited Loading

sophia-guo commented Dec 21, 2024

sophia-guo commented Dec 12, 2024 •

edited

Loading

llxia Dec 13, 2024 •

edited

Loading

sophia-guo Dec 16, 2024 •

edited

Loading

sophia-guo commented Dec 16, 2024 •

edited

Loading

sophia-guo commented Dec 16, 2024 •

edited

Loading

sophia-guo commented Dec 17, 2024 •

edited

Loading