Run Tests times out even though all source-build tests pass

[DonnaChen888]

Repro Pipeline:
20260309.8
20260309.9

Repro Branch:
10.0.3xx
10.0.1xx

Error Message:
##[error]The task has timed out.
Finishing: Run Tests (centOSStreamContainer)

[akoeplinger]

We're also seeing this in public.

Here's a "passing" run from 10.0.1xx where the scenario tests took 22mins: https://dev.azure.com/dnceng-public/public/public%20Team/_build/results?buildId=1326856&view=logs&j=3d049f34-eb7f-5dcd-1393-71cd43c7f766&t=6ba38e20-9330-5100-3c56-0779bf9fce53&l=224

This causes the step to be pretty close to the 30mins timeout. I assume it just runs over that in other cases.

The same test suite took under 2mins just yesterday: https://dev.azure.com/dnceng-public/public/public%20Team/_build/results?buildId=1326488&view=logs&j=3d049f34-eb7f-5dcd-1393-71cd43c7f766&t=6ba38e20-9330-5100-3c56-0779bf9fce53&l=224

[akoeplinger]

Something is causing some tests to take much longer, e.g.

  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: XUnit) [5 m 48 s]  <---- slow
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: Console) [6 s]
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: ClassLib) [6 s]
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: WebApp) [6 s]
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: NUnit) [4 m 41 s]  <---- slow
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: BlazorWasm) [12 m 28 s]  <---- slow

or here

  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: XUnit) [5 m 46 s] <---- slow
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: Console) [7 s]
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: ClassLib) [7 s]
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: WebApp) [7 s]
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: NUnit) [4 m 41 s]  <--- slow
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: BlazorWasm) [1 m 8 s] <--- interestingly faster here
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: Web) [7 s]
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: Worker) [13 s]
  Passed Microsoft.DotNet.SourceBuild.Tests.OmniSharpTests.VerifyScenario(template: MSTest) [8 m 7 s] <!--- slow

[steveisok]

Template	Passing Build	Failing Build	Slowdown
XUnit	14s	5m 45s	25x
NUnit	7s	4m 41s	40x
BlazorWasm	10s	12m 26s	75x
Worker	7s	4m 26s	38x
Console	10s	6s	normal
MSTest	8s	never completed	timed out

[mthalman]

This is not related to any change in the repo. I just reran the test stage in this build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2921880&view=results. It was the build that was fine before the perf regression. Rerunning the test stage has shown the perf issue. So it's something external to the code in the repo.

I'm going to get some better logging in the tests, specifically binlogs, so we can narrow things down.

[steveisok]

Adding some data from investigating the timeout pattern across branches.

The slowdown is template-specific — only templates that pull test framework NuGet packages are affected:

Template	Normal	Slow	Slowdown
XUnit	14s	5m 45s	25x
NUnit	7s	4m 41s	40x
BlazorWasm	10s	12m 26s	75x
MSTest	8s	never completed	timed out
Console	10s	6s	normal
ClassLib	—	7s	normal
WebApp	—	7s	normal

Simple templates (Console, ClassLib, WebApp) run normally every time. The test-framework templates (XUnit, NUnit, MSTest, BlazorWasm) show the 25-75x regression. This points to NuGet restore or package resolution of test framework dependencies as the bottleneck rather than compilation.

Separately, we hit a NuGet restore Too many open files failure on AzureLinux_x64_Cross_arm (dotnet/dotnet#5381) around the same time window, which may indicate NuGet infrastructure stress — though that's a different failure mode.

The timeout workaround PRs (#5354, #5355, #5357) are confirmed working on all 3 release branches — 0 Run Tests failures post-merge. But they don't address root cause.

Binlogs should show where the time is going — my guess is Restore target on the test-framework templates.