Disable Server GC for 32-bit crossgen2/ilc hosts

[jtschuster]

Summary

crossgen2 and ilc enable Server GC unconditionally (AotCompilerCommon.props). When the compiler process itself is 32-bit (e.g. the x86 R2R outerloop leg), Server GC reserves large per-heap segments up front and consumes most of the ~2 GB user-mode address space. Compiling a very large method then fails to allocate the contiguous output-image buffer (MemoryStream.ToArray in EmitChecksums, ~10 MB for HugeArray1.dll) and the process fail-fasts with OutOfMemory — even though resident memory is low.

This switches the AOT compiler host to Workstation GC when the compiler process architecture is 32-bit (x86/arm/armel). The process architecture is CrossHostArch when cross-building, otherwise TargetArchitecture, mirroring the existing TargetArchitectureForSharedLibraries logic. Cross-targeted product builds run crossgen2 as a 64-bit host and keep Server GC, so only the genuinely 32-bit tool process is affected. An explicit ServerGarbageCollection value (CLI/props) still wins.

Fixes #128531.

Why RSS didn't reveal it / how the fix was validated

The bug is virtual address-space reservation, not resident memory, so RSS (VmHWM) can't see it. I measured VmPeak (peak virtual, includes GC mmap reservations) vs VmHWM (resident) for the heaviest compilations on a 32-bit leg:

DLL	Mode	VmPeak (virtual)	VmHWM (resident)	reserved ≈ VmPeak−VmHWM
HugeArray1	Workstation	1362.5 MB	1249.3 MB	113.2 MB
HugeArray1	Server	1572.1 MB	1250.7 MB	321.4 MB
HugeField2	Workstation	923.6 MB	814.8 MB	108.8 MB
HugeField2	Server	1133.3 MB	814.9 MB	318.3 MB

• Resident (VmHWM) is identical across GC modes — the fix costs ~0 real memory.
• Reserved address space drops ~3× (~320 MB → ~110 MB). /proc/<pid>/maps shows Server GC reserving 32–64 MB PROT_NONE (---p) per-heap segments vs 16 MB blocks under Workstation.

On the 2 GB-constrained x86 CI machine (4 CPUs → 4 Server-GC heaps → even larger reservations), that extra reservation plus fragmentation is what exhausts contiguous address space and causes the OOM. Removing it keeps the process under the ceiling.

Note

This pull request was authored with assistance from GitHub Copilot.

[jtschuster]

/azp run runtime-coreclr outerloop

[dotnet-policy-service]

Tagging subscribers to this area: @anicka-net, @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Updates the CoreCLR AOT tool build settings so crossgen2/ilc default to Workstation GC when the compiler process is 32-bit, avoiding large Server-GC virtual address-space reservations that can lead to OOMs in constrained 32-bit address spaces.

Changes:

• Infer the AOT compiler host process architecture from CrossHostArch (when set) or TargetArchitecture.
• Set ServerGarbageCollection only when not explicitly provided, defaulting to true on non-x86/arm/armel hosts and false otherwise.
• Add inline rationale documenting why Server GC is disabled for 32-bit hosts.