Test failure: baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd #109499

v-wenyuxu · 2024-11-04T01:50:30Z

Failed in: runtime-coreclr jitstress-random 20241103.1

Failed tests:

coreclr linux arm Checked jitstress_random_1 @ (Debian.12.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 
Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174477 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
""
System.Exception: Missing "Main" method frame at the last line
   at TestStackOverflow.Program.TestStackOverflowLargeFrameMainThread()
   at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 11/3/2024 7:32:48 PM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /root/helix/work/workitem/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

The text was updated successfully, but these errors were encountered:

v-wenyuxu · 2024-11-05T09:13:38Z

Failed in: runtime-coreclr outerloop 20241104.8

Failed tests:

R2R-CG2 linux x64 Checked no_tiered_compilation @ (Alpine.317.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-amd64
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 DOTNET_DbgEnableMiniDump is set and the createdump binary does not exist: /root/helix/work/correlation/crossgen2/createdump

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
02:36:27
Response file: /root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp
/root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
-o:/root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
--targetarch:x64
--targetos:linux
--verify-type-and-field-layout
--method-layout:random
-r:/root/helix/work/correlation/System.*.dll
-r:/root/helix/work/correlation/Microsoft.*.dll
-r:/root/helix/work/correlation/xunit.*.dll
-r:/root/helix/work/correlation/mscorlib.dll
-r:/root/helix/work/correlation/netstandard.dll
Running CrossGen2:  /root/helix/work/correlation/crossgen2/crossgen2 @/root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp   -r:/root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/*.dll
Emitting R2R PE file: /root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
Running R2RDump:  dotnet /root/helix/work/correlation/R2RDump/R2RDump.dll --header --sc --in /root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll --out /root/helix/work/workitem/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.r2rdump --val
02:36:28
in ReleaseLock
in takeLock
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174386 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"--------------------------------"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(smallframe secondary)
"Stack overflow."
"Repeated 32786 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"
""
Running stackoverflow test(largeframe secondary)
"Stack overflow."
"Repeated 8 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 123
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-06T01:18:29Z

Failed in: runtime-coreclr r2r 20241104.1

Failed tests:

R2R-CG2 linux x64 Checked @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 DOTNET_DbgEnableMiniDump is set and the createdump binary does not exist: /datadisks/disk1/work/99AE0841/p/crossgen2/createdump

cmdLine:/datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 11/5/2024 6:55:03 AM, end: 11/5/2024 7:25:03 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/99AE0841/w/A3660904/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
06:55:03
Response file: /datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp
/datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
-o:/datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
--targetarch:x64
--targetos:linux
--verify-type-and-field-layout
--method-layout:random
-r:/datadisks/disk1/work/99AE0841/p/System.*.dll
-r:/datadisks/disk1/work/99AE0841/p/Microsoft.*.dll
-r:/datadisks/disk1/work/99AE0841/p/xunit.*.dll
-r:/datadisks/disk1/work/99AE0841/p/mscorlib.dll
-r:/datadisks/disk1/work/99AE0841/p/netstandard.dll
Running CrossGen2:  /datadisks/disk1/work/99AE0841/p/crossgen2/crossgen2 @/datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp   -r:/datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/*.dll
Emitting R2R PE file: /datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
Running R2RDump:  dotnet /datadisks/disk1/work/99AE0841/p/R2RDump/R2RDump.dll --header --sc --in /datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll --out /datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.r2rdump --val
06:55:04
in ReleaseLock
in takeLock
/datadisks/disk1/work/99AE0841/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174421 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"

cmdLine:/datadisks/disk1/work/99AE0841/w/A3660904/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 11/5/2024 6:55:03 AM, end: 11/5/2024 7:25:03 AM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/cr

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-07T01:14:30Z

Failed in: runtime-coreclr pgo 20241106.1

Failed tests:

coreclr linux x64 Checked defaultpgo @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 
cmdLine:/datadisks/disk1/work/B2FC0A01/w/C7580A4C/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/6/2024 9:23:30 AM, end: 11/6/2024 9:33:30 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/B2FC0A01/w/C7580A4C/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/B2FC0A01/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174419 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"

cmdLine:/datadisks/disk1/work/B2FC0A01/w/C7580A4C/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/6/2024 9:23:30 AM, end: 11/6/2024 9:33:30 AM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/crashdump_54870.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
----------------------------------
Thread Id: 0xd656
      Child SP               IP Call Site
 0x74d3cf42c260 0x74d3cece57f8 libc.so.6!clock_nanosleep
 0x74d3cf42c2e0 0x74d3cecea677 libc.so.6!__nanosleep
 0x74d3cf42c2f0 0x74d3cecea5ae libc.so.6!sleep
 0x74d3cf42c330 0x74d3ce8918da libcoreclr.so!sigsegv_handler(int, siginfo_t*, void*) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:639:21
 0x74d3cf42c380 0x74d3cec42520 libc.so.6!
 0x7ffda2dfa680 0x74d3508e274e stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffda2dfa690 0x74d3508e271c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffda2e0a6a0 0x74d3508e281c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffda2e1a6b0 0x74d3508e279c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffda2e2a6c0 0x74d3508e271c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffda2e3a6d0 0x74d3508e281c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffda2e4a6e0 0x74d3508e279c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffda2e5a6f0 0x74d3508e271c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffda2e6a700 0x74d3508e281c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffda2e7a710 0x74d3508e279c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffda2e8a720 0x74d3508e271c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffda2e9a730 0x74d3508e281c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffda2eaa740 0x74d3508e279c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffda2eba750 0x74d3508e271c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffda2eca760 0x74d3508e281c stackoverflow.dll!TestStackOverflow.Program.

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-08T01:13:24Z

Failed in: runtime-coreclr crossgen2 20241106.1

Failed tests:

R2R-CG2 linux x64 Checked @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 DOTNET_DbgEnableMiniDump is set and the createdump binary does not exist: /datadisks/disk1/work/BA7E09FE/p/crossgen2/createdump

cmdLine:/datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 11/7/2024 7:27:45 AM, end: 11/7/2024 7:57:45 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/BA7E09FE/w/B9DA099F/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
07:27:45
Response file: /datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp
/datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
-o:/datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
--targetarch:x64
--targetos:linux
--verify-type-and-field-layout
--method-layout:random
-r:/datadisks/disk1/work/BA7E09FE/p/System.*.dll
-r:/datadisks/disk1/work/BA7E09FE/p/Microsoft.*.dll
-r:/datadisks/disk1/work/BA7E09FE/p/xunit.*.dll
-r:/datadisks/disk1/work/BA7E09FE/p/mscorlib.dll
-r:/datadisks/disk1/work/BA7E09FE/p/netstandard.dll
Running CrossGen2:  /datadisks/disk1/work/BA7E09FE/p/crossgen2/crossgen2 @/datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp   -r:/datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/*.dll
Emitting R2R PE file: /datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
Running R2RDump:  dotnet /datadisks/disk1/work/BA7E09FE/p/R2RDump/R2RDump.dll --header --sc --in /datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll --out /datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.r2rdump --val
07:27:45
in ReleaseLock
in takeLock
/datadisks/disk1/work/BA7E09FE/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174423 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"

cmdLine:/datadisks/disk1/work/BA7E09FE/w/B9DA099F/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 11/7/2024 7:27:45 AM, end: 11/7/2024 7:57:45 AM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/cr

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 123
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor14|0_15(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-11T02:01:43Z

Failed in: runtime-coreclr pgostress 20241108.1

Failed tests:

coreclr linux x64 Checked syntheticpgo @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 
cmdLine:/datadisks/disk1/work/A90E0911/w/B5DB09C0/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/9/2024 7:29:28 AM, end: 11/9/2024 7:39:28 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/A90E0911/w/B5DB09C0/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/A90E0911/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174513 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"--------------------------------"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"

cmdLine:/datadisks/disk1/work/A90E0911/w/B5DB09C0/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/9/2024 7:29:28 AM, end: 11/9/2024 7:39:28 AM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/crashdump_19633.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
----------------------------------
Thread Id: 0x4cb1
      Child SP               IP Call Site
 0x720d48df3260 0x720d486e57f8 libc.so.6!clock_nanosleep
 0x720d48df32e0 0x720d486ea677 libc.so.6!__nanosleep
 0x720d48df32f0 0x720d486ea5ae libc.so.6!sleep
 0x720d48df3330 0x720d4829190a libcoreclr.so!sigsegv_handler(int, siginfo_t*, void*) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:641:21
 0x720d48df3380 0x720d48642520 libc.so.6!
 0x7ffdb90e73a0 0x720cca30cbde stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffdb90e73b0 0x720cca30cbac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffdb90f73c0 0x720cca30ccac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffdb91073d0 0x720cca30cc2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffdb91173e0 0x720cca30cbac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffdb91273f0 0x720cca30ccac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffdb9137400 0x720cca30cc2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffdb9147410 0x720cca30cbac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffdb9157420 0x720cca30ccac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffdb9167430 0x720cca30cc2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffdb9177440 0x720cca30cbac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffdb9187450 0x720cca30ccac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffdb9197460 0x720cca30cc2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffdb91a7470 0x720cca30cbac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffdb91b7480 0x720cca30ccac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffdb91c7490 0x720cca30cc2c st

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

AndyAyersMS · 2024-11-11T16:27:32Z

@mangod9 @janvorli we are seeing this failure impact multiple jit pipelines...

janvorli · 2024-11-11T17:01:52Z

In the last four cases something causes the process that hits stack overflow to never exit. The main thread should exit and the secondary threads should wait forever if they hit a stack overflow. There was no change in the related code for months, so it seems like some seemingly unrelated recent change started to trigger this problem.

janvorli · 2024-11-12T16:43:23Z

I have tried to repro it locally on the same Ubuntu with the same settings as the lab used and I was unable to repro it in hundreds of iterations.
The crashreport.json seems to indicate that the sigsegv_handler has believed that other thread is already handling a stack overflow, so it sleeps forever until the process shutdowns. But it never happens, because this test doesn't have stack overflow on other threads.
The wait is only triggered if g_stackOverflowHandlerStack was NULL. It is initialized to a non-null value in SEHInitializeSignals. And if that failed, the PAL initialization would fail and the host would exit before any managed code was executed. Moreover, if it failed, the g_stackOverflowHandlerStack would contain MAP_FAILED, which is non-zero.
And on top of that, the console shows the stack overflow logged to the console, so the main thread must have left the sigsegv handler and executed the runtime stack overflow reporting.
The only remote explanation is that somehow after the stack trace was reported, the runtime ended up returning to the original crash location and re-triggering the stack overflow. But I don't see how that could happen.

v-wenyuxu · 2024-11-13T01:28:49Z

Failed in: runtime-coreclr outerloop 20241112.2

Failed tests:

coreclr linux x64 Checked no_tiered_compilation @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 
cmdLine:/datadisks/disk1/work/A55A0911/w/AF8F09A8/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/12/2024 7:26:52 PM, end: 11/12/2024 7:36:52 PM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/A55A0911/w/AF8F09A8/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/A55A0911/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174502 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"--------------------------------"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(smallframe secondary)
"Stack overflow."
"Repeated 174646 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"
""
Running stackoverflow test(largeframe secondary)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"

cmdLine:/datadisks/disk1/work/A55A0911/w/AF8F09A8/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/12/2024 7:26:52 PM, end: 11/12/2024 7:36:52 PM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/crashdump_53569.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
----------------------------------
Thread Id: 0xd141
      Child SP               IP Call Site
 0x7ffe8294cbf0 0x70a82c091117 libc.so.6!
 0x7ffe8294cc30 0x70a82c093a41 libc.so.6!pthread_cond_wait
 0x7ffe8294cd10 0x70a82bcdeb8b libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:0:28
 0x7ffe8294cd70 0x70a82bcde110 libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:303:22
 0x7ffe8294cdd0 0x70a82bce7b10 libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* co

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-14T01:25:54Z

Failed in: runtime-coreclr outerloop 20241113.1

Failed tests:

R2R-CG2 linux x64 Checked no_tiered_compilation @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 DOTNET_DbgEnableMiniDump is set and the createdump binary does not exist: /datadisks/disk1/work/B59E0A77/p/crossgen2/createdump

cmdLine:/datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 11/13/2024 10:22:57 AM, end: 11/13/2024 10:52:57 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/B59E0A77/w/AC130948/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
10:22:57
Response file: /datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp
/datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
-o:/datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
--targetarch:x64
--targetos:linux
--verify-type-and-field-layout
--method-layout:random
-r:/datadisks/disk1/work/B59E0A77/p/System.*.dll
-r:/datadisks/disk1/work/B59E0A77/p/Microsoft.*.dll
-r:/datadisks/disk1/work/B59E0A77/p/xunit.*.dll
-r:/datadisks/disk1/work/B59E0A77/p/mscorlib.dll
-r:/datadisks/disk1/work/B59E0A77/p/netstandard.dll
Running CrossGen2:  /datadisks/disk1/work/B59E0A77/p/crossgen2/crossgen2 @/datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp   -r:/datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/*.dll
Emitting R2R PE file: /datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
Running R2RDump:  dotnet /datadisks/disk1/work/B59E0A77/p/R2RDump/R2RDump.dll --header --sc --in /datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll --out /datadisks/disk1/work/B59E0A77/w/AC130948/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.r2rdump --val
10:22:58
in ReleaseLock
in takeLock
/datadisks/disk1/work/B59E0A77/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174415 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(smallframe secondary)
"Stack overflow."
"Repeated 174646 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackO

Stack trace:

    at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 123
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-15T01:40:35Z

Failed in: runtime-coreclr crossgen2 20241113.1

Failed tests:

R2R-CG2 linux x64 Checked no_tiered_compilation @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 DOTNET_DbgEnableMiniDump is set and the createdump binary does not exist: /datadisks/disk1/work/A99309E8/p/crossgen2/createdump

cmdLine:/datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 11/14/2024 7:12:34 AM, end: 11/14/2024 7:42:34 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/A99309E8/w/A93109AA/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
07:12:34
Response file: /datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp
/datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
-o:/datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
--targetarch:x64
--targetos:linux
--verify-type-and-field-layout
--method-layout:random
--hot-cold-splitting
-r:/datadisks/disk1/work/A99309E8/p/System.*.dll
-r:/datadisks/disk1/work/A99309E8/p/Microsoft.*.dll
-r:/datadisks/disk1/work/A99309E8/p/xunit.*.dll
-r:/datadisks/disk1/work/A99309E8/p/mscorlib.dll
-r:/datadisks/disk1/work/A99309E8/p/netstandard.dll
Running CrossGen2:  /datadisks/disk1/work/A99309E8/p/crossgen2/crossgen2 @/datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp   -r:/datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/*.dll
Emitting R2R PE file: /datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
Running R2RDump:  dotnet /datadisks/disk1/work/A99309E8/p/R2RDump/R2RDump.dll --header --sc --in /datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll --out /datadisks/disk1/work/A99309E8/w/A93109AA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.r2rdump --val
07:12:34
in ReleaseLock
in takeLock
/datadisks/disk1/work/A99309E8/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174501 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(smallframe secondary)
"Stack overflow."
"Repeated 174646 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 123
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor14|0_15(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-18T01:50:47Z

Failed in: runtime-coreclr pgostress 20241115.1

Failed tests:

coreclr linux x64 Checked syntheticpgo_blend @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 
cmdLine:/datadisks/disk1/work/B5CC09D5/w/B180094E/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/16/2024 7:26:39 AM, end: 11/16/2024 7:36:39 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/B5CC09D5/w/B180094E/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/B5CC09D5/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174529 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"

cmdLine:/datadisks/disk1/work/B5CC09D5/w/B180094E/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/16/2024 7:26:39 AM, end: 11/16/2024 7:36:39 AM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/crashdump_9441.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
----------------------------------
Thread Id: 0x24e1
      Child SP               IP Call Site
 0x72e33c8dc260 0x72e33c2e57f8 libc.so.6!clock_nanosleep
 0x72e33c8dc2e0 0x72e33c2ea677 libc.so.6!__nanosleep
 0x72e33c8dc2f0 0x72e33c2ea5ae libc.so.6!sleep
 0x72e33c8dc330 0x72e33be9367a libcoreclr.so!sigsegv_handler(int, siginfo_t*, void*) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:641:21
 0x72e33c8dc380 0x72e33c242520 libc.so.6!
 0x7ffeba799d70 0x72e2bdf1ce5e stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffeba799d80 0x72e2bdf1ce2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffeba7a9d90 0x72e2bdf1cf2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffeba7b9da0 0x72e2bdf1ceac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffeba7c9db0 0x72e2bdf1ce2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffeba7d9dc0 0x72e2bdf1cf2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffeba7e9dd0 0x72e2bdf1ceac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffeba7f9de0 0x72e2bdf1ce2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffeba809df0 0x72e2bdf1cf2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffeba819e00 0x72e2bdf1ceac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffeba829e10 0x72e2bdf1ce2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffeba839e20 0x72e2bdf1cf2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffeba849e30 0x72e2bdf1ceac stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffeba859e40 0x72e2bdf1ce2c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffeba869e50 0

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 123
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-19T01:21:38Z

Failed in: runtime-coreclr pgo 20241118.1

Failed tests:

coreclr linux x64 Checked defaultpgo @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 
cmdLine:/datadisks/disk1/work/A36008E1/w/A20908D6/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/18/2024 9:26:48 AM, end: 11/18/2024 9:36:48 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/A36008E1/w/A20908D6/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/A36008E1/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174405 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(smallframe secondary)
"Stack overflow."
"Repeated 174646 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"
""
Running stackoverflow test(largeframe secondary)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"
""
Running stackoverflow3 test()
Call number 50 to the Execute method
Call number 100 to the Execute method
Call number 150 to the Execute method
Call number 200 to the Execute method
Call number 250 to the Execute method
Call number 300 to the Execute method
Call number 350 to the Execute method
Call number 400 to the Execute method
Call number 450 to the Execute method
Call number 500 to the Execute method
Call number 550 to the Execute method
Call number 600 to the Execute method
Call number 650 to the Execute method
Call number 700 to the Execute method
Call number 750 to the Execute method
Call number 800 to the Execute method
Call number 850 to the Execute method
Call number 900 to the Execute method
Call number 950 to the Execute method
Call number 1000 to the Execute method
"Stack overflow."
"Repeated 1033 times:"
"--------------------------------"
"   at TestStackOverflow3.Program.Execute(System.String)"
"--------------------------------"
"   at TestStackOverflow3.Program.Main()"

cmdLine:/datadisks/disk1/work/A36008E1/w/A20908D6/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variab

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-20T01:13:01Z

Failed in: runtime-coreclr crossgen2 20241118.1

Failed tests:

R2R-CG2 linux x64 Checked @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 DOTNET_DbgEnableMiniDump is set and the createdump binary does not exist: /datadisks/disk1/work/A3C708CF/p/crossgen2/createdump

cmdLine:/datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 11/19/2024 7:09:58 AM, end: 11/19/2024 7:39:58 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/A3C708CF/w/B24609BA/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
07:09:58
Response file: /datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp
/datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/stackoverflowtester.dll
-o:/datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
--targetarch:x64
--targetos:linux
--verify-type-and-field-layout
--method-layout:random
-r:/datadisks/disk1/work/A3C708CF/p/System.*.dll
-r:/datadisks/disk1/work/A3C708CF/p/Microsoft.*.dll
-r:/datadisks/disk1/work/A3C708CF/p/xunit.*.dll
-r:/datadisks/disk1/work/A3C708CF/p/mscorlib.dll
-r:/datadisks/disk1/work/A3C708CF/p/netstandard.dll
Running CrossGen2:  /datadisks/disk1/work/A3C708CF/p/crossgen2/crossgen2 @/datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.rsp   -r:/datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/IL-CG2/*.dll
Emitting R2R PE file: /datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll
Running R2RDump:  dotnet /datadisks/disk1/work/A3C708CF/p/R2RDump/R2RDump.dll --header --sc --in /datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll --out /datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.dll.r2rdump --val
07:09:59
in ReleaseLock
in takeLock
/datadisks/disk1/work/A3C708CF/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174438 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"--------------------------------"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"

cmdLine:/datadisks/disk1/work/A3C708CF/w/B24609BA/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 11/19/2024 7:09:58 AM, end: 11/19/2024 7:39:58 AM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/crashdump_84534.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
-----------------

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 123
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor14|0_15(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu · 2024-11-21T01:35:09Z

Failed in: runtime-coreclr outerloop 20241120.2

Failed tests:

coreclr linux x64 Checked no_tiered_compilation @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 
cmdLine:/datadisks/disk1/work/B61909D9/w/A4DD0919/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/20/2024 7:36:29 PM, end: 11/20/2024 7:46:29 PM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/B61909D9/w/A4DD0919/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/B61909D9/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174492 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"--------------------------------"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(smallframe secondary)
"Stack overflow."
"Repeated 174646 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"
""
Running stackoverflow test(largeframe secondary)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program+<>c__DisplayClass7_0.<SecondaryThreadsTest>b__0()"

cmdLine:/datadisks/disk1/work/B61909D9/w/A4DD0919/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/20/2024 7:36:29 PM, end: 11/20/2024 7:46:29 PM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/crashdump_16889.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
----------------------------------
Thread Id: 0x41f9
      Child SP               IP Call Site
 0x7ffe42aaa830 0x75cc3ac91117 libc.so.6!
 0x7ffe42aaa870 0x75cc3ac93a41 libc.so.6!pthread_cond_wait
 0x7ffe42aaa950 0x75cc3a8e02cb libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:0:28
 0x7ffe42aaa9b0 0x75cc3a8df850 libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:303:22
 0x7ffe42aaaa10 0x75cc3a8e9250 libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* co

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 141
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

janvorli · 2024-11-21T21:49:06Z

I still cannot repro the issue locally, but I have looked at couple of the logs from the failing runs above and I've noticed that in some of those part of the stack trace shows the following:

0x7c2039bdd260 0x7c2039ce57f8 libc.so.6!clock_nanosleep
 0x7c2039bdd2e0 0x7c2039cea677 libc.so.6!__nanosleep
 0x7c2039bdd2f0 0x7c2039cea5ae libc.so.6!sleep
 0x7c2039bdd330 0x7c2039891c2a libcoreclr.so!sigsegv_handler(int, siginfo_t*, void*) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:641:21
 0x7c2039bdd380 0x7c2039c42520 libc.so.6!
 0x7c203a2aeb00 0x7c20398b3aaa libcoreclr.so!DBG_printf at /__w/1/s/src/coreclr/pal/src/misc/dbgmsg.cpp:455:0
 0x7c203a2b3aa0 0x7c20398f60e7 libcoreclr.so!AssertBreak at /__w/1/s/src/coreclr/pal/src/include/pal/dbgmsg.h:346:8
 (inlined by) SetThreadDescription at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1492:21
 0x7c203a2b3b20 0x7c20393fd196 libcoreclr.so!Thread::CreateUtilityThread(Thread::StackSizeBucket, unsigned int (*)(void*), void*, char16_t const*, unsigned int, unsigned int*) at /__w/1/s/src/coreclr/vm/threads.cpp:2020:9
 0x7c203a2b3b70 0x7c2039498004 libcoreclr.so!HolderBase<void*>::HolderBase(void*) at /__w/1/s/src/coreclr/inc/holder.h:120:9
 (inlined by) FunctionBase<void*, &(void DoNothing<void*>(void*)), &(VoidCloseHandle(void*))>::FunctionBase(void*) at /__w/1/s/src/coreclr/inc/holder.h:701:9
 (inlined by) BaseHolder<void*, FunctionBase<void*, &(void DoNothing<void*>(void*)), &(VoidCloseHandle(void*))>, 18446744073709551615ul, &(int CompareDefault<void*>(void*, void*))>::BaseHolder(void*) at /__w/1/s/src/coreclr/inc/holder.h:242:9
 (inlined by) BaseWrapper<void*, FunctionBase<void*, &(void DoNothing<void*>(void*)), &(VoidCloseHandle(void*))>, 18446744073709551615ul, &(int CompareDefault<void*>(void*, void*))>::BaseWrapper(void*) at /__w/1/s/src/coreclr/inc/holder.h:573:11
 (inlined by) Wrapper<void*, &(void DoNothing<void*>(void*)), &(VoidCloseHandle(void*)), 18446744073709551615ul, &(int CompareDefault<void*>(void*, void*)), true>::Wrapper(void*) at /__w/1/s/src/coreclr/inc/holder.h:791:11
 (inlined by) EEPolicy::HandleFatalStackOverflow(_EXCEPTION_POINTERS*, int) at /__w/1/s/src/coreclr/vm/eepolicy.cpp:668:46

It indicates that when we were creating a helper thread to report the stack overflow, the SetThreadDescription has asserted. This failure comes from the _ASSERTE here:

runtime/src/coreclr/pal/src/thread/thread.cpp

Lines 1490 to 1492 in 09b30a4

    
           int setNameResult = minipal_set_thread_name(pTargetThread->GetPThreadSelf(), nameBuf); 
        
           (void)setNameResult; // used 
        
           _ASSERTE(setNameResult == 0);

This _ASSERTE was added in #109470 and the current issue started to occur at the same day it went in. While I don't know why would the minipal_set_thread_name fail, it seems to be the culprit here.

Since the DBG_Printf called by the _ASSERTE has crashed with sigsegv and the line number indicates it was at the first line of that function, it is possible that the special stack that we use to execute the stack overflow handling is no longer sufficient due to some data structures growth.

I am going to try to add one more page to that space and see if the problem goes away.

janvorli · 2024-11-21T22:53:32Z

I was actually able to repro it by running the stackoverflow test under LLDB. The pthread_setname_np that the minipal_set_thread_name invokes really fails with ENOENT. And that happens because the thread creation in the code below fails.

runtime/src/coreclr/vm/threads.cpp

Lines 2016 to 2023 in 09b30a4

    
           HANDLE hThread = CreateThread(NULL, stackSize, start, args, flags, &threadId); 
        
           SetThreadName(hThread, pName); 
        
           if (pThreadId) 
        
               *pThreadId = threadId; 
        
           return hThread;

And as you can see, the call to SetThreadName is invoked even if the CreateThread returns INVALID_HANDLE_VALUE indicating it has failed. The pthread_setname_np then fails as the thread doesn't exist.

And finally, the thread creation fails due to the stack space of the stack overflow helper stack not being sufficient anymore, as I have thought. Adding one more page to this helper stack fixed the problem.

The stack overflow coreclr tests started to fail recently. It turns out that was caused by the size of the helper stack allocated for stack overflow handling case is no longer sufficient. Moreover, there is a bug in `Thread::CreateUtilityThread` that calls the `SetThreadName` even when the thread creation fails. Close dotnet#109499

v-wenyuxu · 2024-11-22T02:05:06Z

Failed in: runtime-coreclr pgo 20241121.1

Failed tests:

coreclr linux x64 Checked defaultpgo @ Ubuntu.2204.Amd64.Open
    - baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd

Error message:

 
cmdLine:/datadisks/disk1/work/AD6D0971/w/AFA209EB/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/21/2024 9:22:48 AM, end: 11/21/2024 9:32:48 AM)

Return code:      -100
Raw output file:      /datadisks/disk1/work/AD6D0971/w/AFA209EB/uploads/stackoverflow/stackoverflowtester/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/AD6D0971/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true stackoverflowtester.dll ''
Running stackoverflow test(smallframe main)
"Stack overflow."
"Repeated 174403 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionC()"
"   at TestStackOverflow.Program.InfiniteRecursionB()"
"   at TestStackOverflow.Program.InfiniteRecursionA()"
"--------------------------------"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"
""
Running stackoverflow test(largeframe main)
"Stack overflow."
"Repeated 42 times:"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.InfiniteRecursionC2()"
"--------------------------------"
"   at TestStackOverflow.Program.InfiniteRecursionB2()"
"   at TestStackOverflow.Program.InfiniteRecursionA2()"
"   at TestStackOverflow.Program.Test(Boolean)"
"   at TestStackOverflow.Program.Main(System.String[])"

cmdLine:/datadisks/disk1/work/AD6D0971/w/AFA209EB/e/baseservices/exceptions/baseservices-exceptions/../stackoverflow/stackoverflowtester/stackoverflowtester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 11/21/2024 9:22:48 AM, end: 11/21/2024 9:32:48 AM)
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/crashdump_50118.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
----------------------------------
Thread Id: 0xc3c6
      Child SP               IP Call Site
 0x757acd5de260 0x757accee57f8 libc.so.6!clock_nanosleep
 0x757acd5de2e0 0x757acceea677 libc.so.6!__nanosleep
 0x757acd5de2f0 0x757acceea5ae libc.so.6!sleep
 0x757acd5de330 0x757acca934da libcoreclr.so!sigsegv_handler(int, siginfo_t*, void*) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:641:21
 0x757acd5de380 0x757acce42520 libc.so.6!
 0x7ffc975e5780 0x757a4eaa289e stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffc975e5790 0x757a4eaa286c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffc975f57a0 0x757a4eaa29dc stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffc976057b0 0x757a4eaa28ec stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffc976157c0 0x757a4eaa286c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffc976257d0 0x757a4eaa29dc stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffc976357e0 0x757a4eaa28ec stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffc976457f0 0x757a4eaa286c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffc97655800 0x757a4eaa29dc stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffc97665810 0x757a4eaa28ec stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffc97675820 0x757a4eaa286c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffc97685830 0x757a4eaa29dc stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffc97695840 0x757a4eaa28ec stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionB2()
 0x7ffc976a5850 0x757a4eaa286c stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionA2()
 0x7ffc976b5860 0x757a4eaa29dc stackoverflow.dll!TestStackOverflow.Program.InfiniteRecursionC2()
 0x7ffc976c5870 0x757a4eaa28e

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor134|0_135(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu added arch-arm32 blocking-clean-ci-optional Blocking optional rolling runs JitStress CLR JIT issues involving JIT internal stress modes os-linux Linux OS (any supported distro) labels Nov 4, 2024

dotnet-issue-labeler bot added the area-ExceptionHandling-coreclr label Nov 4, 2024

dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Nov 4, 2024

amanasifkhalid mentioned this issue Nov 19, 2024

JIT: Remove switch recognition fallthrough quirk #109796

Merged

janvorli linked a pull request Nov 21, 2024 that will close this issue

Enlarge stack overflow handling helper stack #110068

Open

dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Nov 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test failure: baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd #109499

Test failure: baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd #109499

v-wenyuxu commented Nov 4, 2024

v-wenyuxu commented Nov 5, 2024

v-wenyuxu commented Nov 6, 2024

v-wenyuxu commented Nov 7, 2024

v-wenyuxu commented Nov 8, 2024

v-wenyuxu commented Nov 11, 2024

AndyAyersMS commented Nov 11, 2024

janvorli commented Nov 11, 2024

janvorli commented Nov 12, 2024

v-wenyuxu commented Nov 13, 2024

v-wenyuxu commented Nov 14, 2024

v-wenyuxu commented Nov 15, 2024

v-wenyuxu commented Nov 18, 2024

v-wenyuxu commented Nov 19, 2024

v-wenyuxu commented Nov 20, 2024

v-wenyuxu commented Nov 21, 2024

janvorli commented Nov 21, 2024

janvorli commented Nov 21, 2024

v-wenyuxu commented Nov 22, 2024

Test failure: baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd #109499

Test failure: baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd #109499

Comments

v-wenyuxu commented Nov 4, 2024

v-wenyuxu commented Nov 5, 2024

v-wenyuxu commented Nov 6, 2024

v-wenyuxu commented Nov 7, 2024

v-wenyuxu commented Nov 8, 2024

v-wenyuxu commented Nov 11, 2024

AndyAyersMS commented Nov 11, 2024

janvorli commented Nov 11, 2024

janvorli commented Nov 12, 2024

v-wenyuxu commented Nov 13, 2024

v-wenyuxu commented Nov 14, 2024

v-wenyuxu commented Nov 15, 2024

v-wenyuxu commented Nov 18, 2024

v-wenyuxu commented Nov 19, 2024

v-wenyuxu commented Nov 20, 2024

v-wenyuxu commented Nov 21, 2024

janvorli commented Nov 21, 2024

janvorli commented Nov 21, 2024

v-wenyuxu commented Nov 22, 2024