Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update MS VS ReDist DLLs distributed to "latest stable" #1104

Closed
andrew-m-leonard opened this issue Sep 4, 2024 · 20 comments
Closed

Update MS VS ReDist DLLs distributed to "latest stable" #1104

andrew-m-leonard opened this issue Sep 4, 2024 · 20 comments
Assignees
Labels
testing x64 Issues that affect or relate to the x64/x32 LINUX OS

Comments

@andrew-m-leonard
Copy link
Contributor

Recent issue highlighted we currently ship an old UCRT DLL that can cause in-compatibilities with certain API calls with the latest Win SDK.
We should move up to a more recent "fixed" stable set, we should also specify the complete set so we are consistent:
Currently this, with the additional DLLs loaded from the installed VS layout:

--with-ucrt-dll-dir='C:/progra~2/wi3cf2~1/10/Redist/10.0.22621.0/ucrt/DLLs/x64'

Should update to use the complete set:

--with-msvcr-dll: points to the VS C runtime main DLL
--with-vcruntime-1-dll: points to the VS C runtime extra DLL
--with-msvcp-dll: points to the VS C++ runtime main DLL
--with-ucrt-dll-dir: point to the directory for the Universal C runtime DLLs (all of them)
@github-actions github-actions bot added testing x64 Issues that affect or relate to the x64/x32 LINUX OS labels Sep 4, 2024
@andrew-m-leonard andrew-m-leonard self-assigned this Sep 23, 2024
@andrew-m-leonard
Copy link
Contributor Author

Possibly update to latest:
image

@andrew-m-leonard
Copy link
Contributor Author

@andrew-m-leonard
Copy link
Contributor Author

The full historic list of MSVC++ version are here: https://en.wikipedia.org/wiki/Microsoft_Visual_C++
The current is:

Visual Studio 2022 version 17.10.1 	17.10.1 	2022 	1940 	194033811 	14.40.33811 

@andrew-m-leonard
Copy link
Contributor Author

The latest MSVC Redist files can be downloaded as installer executable file here: https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

When vc_redist.x64.exe will install the redist DLLs into the c:\Windows\System32 directory as long as there isn't a later version of the redist in that folder already.

To package a set of redist DLL's necessary to package with Temurin, I would suggest the following process:

  1. Determine the set of redist files that are packaged with Temurin
  2. On a suitable Windows VM ensure the c:\Windows\System32 directory has all the redist DLLs removed
  3. Then download the required redist version from https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170
  4. Run the redist.exe, which will extract the DLLs into the c:\Windows\System32 directory
  5. Zip the necessary redist DLLs into a zip file and store...

For building Temurin at ci.adoptium.net:

  1. Extract the "stored" redist files into a suitable local directory
  2. Configure the openjdk make with necessary configure options pointing at the extract redist local directory:
--with-msvcr-dll: points to the VS C runtime main DLL
--with-vcruntime-1-dll: points to the VS C runtime extra DLL
--with-msvcp-dll: points to the VS C++ runtime main DLL
--with-ucrt-dll-dir: point to the directory for the Universal C runtime DLLs (all of them)

@andrew-m-leonard
Copy link
Contributor Author

@sxa @steelhead31 Your thoughts please, if we manually package the latest "redist" DLLs, where could we put them for our build nodes?

@steelhead31
Copy link
Contributor

@andrew-m-leonard I think we could, package up the zip files , and have them deployed by Ansible into a specifie folder on the windows build machines, we could approach this in a way to allow multiple versions, or alternatively just a single version... something along the lines of..

c:\temurin-msvc-redist\version1.7.0
c:\temurin-msvc-redist\version2.7.0
c:\temurin-msvc-redist\latest
..etc..etc..

Could then define the locations for the parameters listed per version as required ?

@andrew-m-leonard
Copy link
Contributor Author

@andrew-m-leonard I think we could, package up the zip files , and have them deployed by Ansible into a specifie folder on the windows build machines, we could approach this in a way to allow multiple versions, or alternatively just a single version... something along the lines of..

c:\temurin-msvc-redist\version1.7.0 c:\temurin-msvc-redist\version2.7.0 c:\temurin-msvc-redist\latest ..etc..etc..

Could then define the locations for the parameters listed per version as required ?

Thanks Scott, yes that sounds reasonable, I will create the latest Redist version package (17.10 - 14.40.33810)

@andrew-m-leonard
Copy link
Contributor Author

The Microsoft Redist "update" package installer seems to only install the vcruntime*, msvcp*.dll and ucrtbase.dll, all of which are put into c:\Windows\System32 when you run say the vc_Redist_x64.exe. It does not install the api-ms-crt*.dll's which seem only present in the Windows Kit/Redist/ucrt/DLLs folder.

Trying to accurately create our own Redist.zip packaging x64,arm64,x86 and all the necessary dlls is going to be tricky, and probably not the right thing to do.

We should probably instead create a "Windows DevKit" for VS2022 using: https://github.com/openjdk/jdk/blob/master/make/devkit/createWindowsDevkit.sh
This requires the "required" versions of VS2022 and Windows Kit SDK to be installed on your machine, it will then create a "DevKit" package from them.

We should thus look to update to the latest(possibly LTSC) MS VS2022 update from: https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-history
and create a "Windows DevKit" from it.

@andrew-m-leonard
Copy link
Contributor Author

Creating and using a updated Windows VS2022 "DevKit" would then resolve issue: adoptium/temurin-build#3887

@andrew-m-leonard andrew-m-leonard moved this from Todo to In Progress in 2024 3Q Adoptium Plan Oct 1, 2024
@andrew-m-leonard
Copy link
Contributor Author

A Windows DevKit would require further investigation and testing.
To alleviate the user issues found by adoptium/temurin-build#3887
I am going to try a build test using the latest VS2022 17.10 Redist for just the C and C++ "runtime" DLLs:

--with-msvcr-dll: points to the VS C runtime main DLL
--with-vcruntime-1-dll: points to the VS C runtime extra DLL
--with-msvcp-dll: points to the VS C++ runtime main DLL

@fthevenet
Copy link

fthevenet commented Oct 1, 2024

Hi Andrew

In order to get the UCRT dll to redistribute with the jdk, it seems we have to get them from the Windows SDK:
https://devblogs.microsoft.com/cppblog/introducing-the-universal-crt/#distributing-software-that-uses-the-universal-crt (see point 6: "App-local deployment of the Universal CRT is supported")

The "easiest" way to extract a complete collection for these DLLs, is to proceed as follow:

  1. Download an ISO image of the latest SDK from this location: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/
  2. Expand the ISO image to disk
  3. Locate the installation bundle "Universal CRT Redistributable-x86_en-us.msi" and de-compile it using the "dark.exe" MSI de-compiler that comes with the Wix Toolkit. I.e.
c:\Tools\wix311\bin\dark.exe -v -o c:\sdk\out\ucrt.wxs  "c:\sdk\26100.1742.240904-1906.ge_release_svc_prod1_WindowsSDK\Installers\Universal CRT Redistributable-x86_en-us.msi" -x c:\sdk\out
  1. The DLLs binaries will have been extracted in "C:\sdk\out\File", but the files are named using the their msi IDs
  2. The Wix sources generated by the de-compiler in c:\sdk\out\ucrt.wxs contains the correspondence between each ID and the dll names, so it should be easy to write a script that renames them.

While this isn't exactly "easy" (I said "easiest"...), doing things this way presents several advantages over having to run the installer on an (hopefully) clean machine and fish out what we believe are the freshly installed files, mainly that I find it easier to trust we have the extracted the right files. It might also be simpler to automate the process.

EDIT: Actually, unlike the VC_redist bundle for the legacy C runtime which dumps everything in system32, running "Universal CRT Redistributable-x86_en-us.msi" installed the DLL in an easily identifiable location in C:\Program Files (x86)\Windows Kits\10\Redist, and it does not overwrite previous versions. So afterall, it might be simpler to just get the files directly from there and avoid having to deal with dark magic.

@fthevenet
Copy link

By the way, you can use dark.exe in a similar way to extract the legacy VC runtime DLLs from the VC_redist bundle.

@andrew-m-leonard
Copy link
Contributor Author

Hi Andrew

In order to get the UCRT dll to redistribute with the jdk, it seems we have to get them from the Windows SDK: https://devblogs.microsoft.com/cppblog/introducing-the-universal-crt/#distributing-software-that-uses-the-universal-crt (see point 6: "App-local deployment of the Universal CRT is supported")

The "easiest" way to extract a complete collection for these DLLs, is to proceed as follow:

1. Download an ISO image of the latest SDK from this location: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/

2. Expand the ISO image to disk

3. Locate the installation bundle "Universal CRT Redistributable-x86_en-us.msi" and de-compile it using the "dark.exe" MSI de-compiler that comes with the Wix Toolkit. I.e.
c:\Tools\wix311\bin\dark.exe -v -o c:\sdk\out\ucrt.wxs  "c:\sdk\26100.1742.240904-1906.ge_release_svc_prod1_WindowsSDK\Installers\Universal CRT Redistributable-x86_en-us.msi" -x c:\sdk\out
4. The DLLs binaries will have been extracted in "C:\sdk\out\File", but the files are named using the their msi IDs

5. The Wix sources generated by the de-compiler in c:\sdk\out\ucrt.wxs  contains the correspondence between each ID and the dll names, so it should be easy to write a script that renames them.

While this isn't exactly "easy" (I said "easiest"...), doing things this way presents several advantages over having to run the installer on an (hopefully) clean machine and fish out what we believe are the freshly installed files, mainly that I find it easier to trust we have the extracted the right files. It might also be simpler to automate the process.

EDIT: Actually, unlike the VC_redist bundle for the legacy C runtime which dumps everything in system21, running "Universal CRT Redistributable-x86_en-us.msi" installed the DLL in an easily identifiable location in C:\Program Files (x86)\Windows Kits\10\Redist, and it does not overwrite previous versions. So afterall, it might be simpler to just get the files directly from there and avoid having to deal with dark magic.

Thanks @fthevenet, for the methods. For the linked issue we believe we just need to update the C runtime DLLs only. so I am running a test with just those at the moment

@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Oct 3, 2024

To extract the required Redist DLLs, the safest and easiest way is to locally install an "all architecture" installation as follows:

  1. Determine the "Build Tools" version install bootstrapper required and download from here: https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-history#fixed-version-bootstrappers
  2. Install locally Visual Studio on a Windows x64 VM using the following command:
./vs_BuildTools.exe --passive --norestart --wait --arch all --add "Microsoft.VisualStudio.Workload.NativeDesktop;includeRecommended;includeOptional" --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.VC.Tools.ARM64 --add Microsoft.VisualStudio.Component.VC.ATL.ARM64 --add Microsoft.VisualStudio.Component.VC.MFC.ARM64
  1. Determine the required version of the "Windows SDK" from here: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/
  • Note the point version, eg."10.0.26100.1742"
  • Download the "Installer" from this page
  1. Run the Windows SDK Installer locally to install the required Windows SDK Redist UCRT DLLs

Now extract the required "Redist" DLLs as follows:

  1. Find the correct MSVC Redist folders, it should be located under folder:
C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Redist\MSVC\xx.yy.zzzzz\<arch>\Microsoft.VC143.CRT
  1. Copy the following MSVC Redist DLLs for each architecture (arm64, x64, x86):
vcruntime140.dll
vcruntime140_1.dll (not x86)
msvcp140.dll
  1. Find the correct Windows Kit UCRT folder, it should be located under folder:
C:\Program Files (x86)\Windows Kits\10\Redist\10.0.xxxxx.0\ucrt
  1. Copy the entire "ucrt" sub-folder containing the DLLs/(arm64, x64, x86), eg:
cp -r "Program Files (x86)/Windows Kits/10/Redist/10.0.26100.0/ucrt" .

@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Oct 3, 2024

I have created a Redist update using the above method, for the following versions:

  • Visual Studio 2022 - LTSC 17.10 : 17.10.7 : September 10, 2024 Build Tools : 17.10.35230.96
  • Including "Windows 11 SDK (10.0.26100)"

The zip has been uploaded to the devkit_binaries : https://github.com/adoptium/devkit-binaries/releases/tag/vs2022_redist_14.40.33807_10.0.26100.0

@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Oct 4, 2024

Test builds running:

x64:
jdk8u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-temurin/514/

09:46:25  	MiniMix_5m_0
09:46:25  	MiniMix_5m_1
09:46:25  	MiniMix_10m_0
09:46:25  	MiniMix_10m_1
09:46:25  	MiniMix_aot_5m_0

jdk11u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-temurin/404/ - Success
jdk17u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-windows-x64-temurin/531/ - Success
jdk21u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-windows-x64-temurin/165/ - Success
jdk23u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23u/job/jdk23u-windows-x64-temurin/1/ - Success

x86:
jdk8u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x86-32-temurin/491/

09:42:35  	MiniMix_5m_2
09:42:35  	MiniMix_10m_2
09:45:30  	MauveSingleThrdLoad_HS_5m_2
09:45:30  	MauveSingleInvocLoad_HS_5m_2
09:45:30  	MauveMultiThrdLoad_5m_2

jdk11u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x86-32-temurin/375/

21:17:44   stdout: [[0.026s][info][cds] _jvm_ident expected: OpenJDK Client VM (11.0.25-beta+8-202410071436) for windows-x86 JRE (11.0.25-beta+8-202410071436), built on Oct  7 2024 15:00:43 by "" with MS VC++ 17.7 (VS2022)
21:17:44  [0.026s][info][cds]              actual: OpenJDK Server VM (11.0.25-beta+8-202410071436) for windows-x86 JRE (11.0.25-beta+8-202410071436), built on Oct  7 2024 15:02:05 by "" with MS VC++ 17.7 (VS2022)
21:17:44  [0.026s][info][cds] UseSharedSpaces: The shared archive file was created by a different version or build of HotSpot

jdk17u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-windows-x86-32-temurin/363/

09:57:31  			TEST: runtime/cds/TestCDSVMCrash.java

aarch64:
jdk21u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-windows-aarch64-temurin/3/

TEST: java/lang/Thread/virtual/stress/PingPong.java#ltq
09:55:05  			TEST: runtime/cds/CheckDefaultArchiveFile.java
09:55:05          TEST: runtime/cds/TestCDSVMCrash.java
09:55:05          TEST: runtime/ErrorHandling/UncaughtNativeExceptionTest.java
09:55:05          TEST: serviceability/sa/ClhsdbFindPC.java#no-xcomp-core
09:55:05          TEST: serviceability/sa/ClhsdbDumpheap.java
09:55:05          TEST: serviceability/sa/ClhsdbFindPC.java#no-xcomp-process
09:55:05          TEST: serviceability/sa/ClhsdbFindPC.java#xcomp-core
09:55:05          TEST: serviceability/sa/ClhsdbCDSJstackPrintAll.java
09:55:05          TEST: serviceability/sa/ClhsdbJdis.java
09:55:05          TEST: serviceability/sa/ClhsdbJstack.java#id1
09:55:05          TEST: serviceability/sa/ClhsdbPmap.java#core
09:55:05          TEST: serviceability/sa/ClhsdbPrintAs.java
09:55:05          TEST: serviceability/sa/ClhsdbPstack.java#core
09:55:05          TEST: serviceability/sa/ClhsdbSource.java
09:55:05          TEST: serviceability/sa/ClhsdbThreadContext.java
09:55:05          TEST: serviceability/sa/ClhsdbWhere.java
09:55:05          TEST: serviceability/sa/TestJhsdbJstackLock.java
09:55:05          TEST: serviceability/sa/TestClhsdbJstackLock.java

jdk23u https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23u/job/jdk23u-windows-aarch64-temurin/1/

00:03:36  FAILED test targets:
00:03:36  	hotspot_tier1_runtime_0 - Test results: passed: 605; failed: 6 
00:03:36  		Failed test cases: 
00:03:36  			TEST: runtime/cds/CheckDefaultArchiveFile.java
00:03:36          TEST: runtime/cds/TestCDSVMCrash.java
00:03:36          TEST: runtime/ErrorHandling/MachCodeFramesInErrorFile.java
00:03:36          TEST: runtime/ErrorHandling/UncaughtNativeExceptionTest.java
00:03:36          TEST: runtime/memory/ReadFromNoaccessArea.java
00:03:36          TEST: runtime/jni/nativeStack/TestNativeStack.java
00:03:36          
00:03:36  	hotspot_tier1_serviceability_0 - Test results: passed: 328; failed: 16 
00:03:36  		Failed test cases: 
00:03:36  			TEST: serviceability/jvmti/RedefineClasses/RedefineSharedClass.java
00:03:36          TEST: serviceability/sa/ClhsdbCDSJstackPrintAll.java
00:03:36          TEST: serviceability/sa/ClhsdbDumpheap.java
00:03:36          TEST: serviceability/sa/ClhsdbFindPC.java#no-xcomp-core
00:03:36          TEST: serviceability/sa/ClhsdbFindPC.java#no-xcomp-process
00:03:36          TEST: serviceability/sa/ClhsdbFindPC.java#xcomp-core
00:03:36          TEST: serviceability/sa/ClhsdbJdis.java
00:03:36          TEST: serviceability/sa/ClhsdbJstack.java#id1
00:03:36          TEST: serviceability/sa/ClhsdbPmap.java#core
00:03:36          TEST: serviceability/sa/ClhsdbPrintAs.java
00:03:36          TEST: serviceability/sa/ClhsdbPstack.java#core
00:03:36          TEST: serviceability/sa/ClhsdbSource.java
00:03:36          TEST: serviceability/sa/ClhsdbThreadContext.java
00:03:36          TEST: serviceability/sa/ClhsdbWhere.java
00:03:36          TEST: serviceability/sa/TestClhsdbJstackLock.java
00:03:36          TEST: serviceability/sa/TestJhsdbJstackLock.java
00:03:36          
00:03:36  	jdk_lang_0 - Test results: passed: 999; error: 2 
00:03:36  		Failed test cases: 
00:03:36  			TEST: java/lang/Thread/virtual/stress/PingPong.java#ltq
00:03:36          TEST: java/lang/Thread/virtual/stress/PingPong.java#sq
00:03:36          
00:03:36  	jdk_lang_1 - Test results: passed: 1,000; error: 1 
00:03:36  		Failed test cases: 
00:03:36  			TEST: java/lang/Thread/virtual/stress/PingPong.java#ltq
00:03:36          
00:03:36  	jdk_util_1 - Test results: passed: 988; error: 1 
00:03:36  		Failed test cases: 
00:03:36  			TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java

@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Oct 8, 2024

Re-running these 3 failures with a longer timeout of x50:

TEST: java/lang/Thread/virtual/stress/PingPong.java#sq
TEST: java/lang/Thread/virtual/stress/PingPong.java#ltq
TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java

https://ci.adoptium.net/job/Grinder/11135/
Failed, PingPong only managed 235k iterations out of the 10,000,000 default

Re-running with just TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java :
https://ci.adoptium.net/job/Grinder/11136/ - Success 10mins

@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Oct 8, 2024

Mauve related tests that nearly always fail in jdk8u:

09:46:25  	MiniMix_5m
09:46:25  	MiniMix_10m
09:46:25  	MiniMix_aot_5m_0
09:45:30  	MauveSingleThrdLoad_HS_5m
09:45:30  	MauveSingleInvocLoad_HS_5m
09:45:30  	MauveMultiThrdLoad_5m

@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Oct 8, 2024

Newly added hotspot_tier1_runtime_0 and hotspot_tier1_serviceability_0 (adoptium/aqa-tests#5648), that always fail:

hotspot_tier1_runtime:

TEST: runtime/CDSCompressedKPtrs/XShareAuto.java
TEST: runtime/cds/TestCDSVMCrash.java
TEST: runtime/cds/CheckDefaultArchiveFile.java
TEST: runtime/ErrorHandling/UncaughtNativeExceptionTest.java
TEST: runtime/ErrorHandling/MachCodeFramesInErrorFile.java
TEST: runtime/memory/ReadFromNoaccessArea.java
TEST: runtime/jni/nativeStack/TestNativeStack.java

hotspot_tier1_serviceability:

TEST: serviceability/sa/ClhsdbFindPC.java#no-xcomp-core
TEST: serviceability/sa/ClhsdbDumpheap.java
TEST: serviceability/sa/ClhsdbFindPC.java#no-xcomp-process
TEST: serviceability/sa/ClhsdbFindPC.java#xcomp-core
TEST: serviceability/sa/ClhsdbCDSJstackPrintAll.java
TEST: serviceability/sa/ClhsdbJdis.java
TEST: serviceability/sa/ClhsdbJstack.java#id1
TEST: serviceability/sa/ClhsdbPmap.java#core
TEST: serviceability/sa/ClhsdbPrintAs.java
TEST: serviceability/sa/ClhsdbPstack.java#core
TEST: serviceability/sa/ClhsdbSource.java
TEST: serviceability/sa/ClhsdbThreadContext.java
TEST: serviceability/sa/ClhsdbWhere.java
TEST: serviceability/sa/TestJhsdbJstackLock.java
TEST: serviceability/sa/TestClhsdbJstackLock.java
TEST: serviceability/jvmti/RedefineClasses/RedefineSharedClass.java

Raised: adoptium/aqa-tests#5671

@andrew-m-leonard
Copy link
Contributor Author

Resolved by adoptium/temurin-build#3981

@github-project-automation github-project-automation bot moved this from In Progress to Done in 2024 4Q Adoptium Plan Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing x64 Issues that affect or relate to the x64/x32 LINUX OS
Projects
Status: Done
Development

No branches or pull requests

3 participants