Skip to content

Conversation

@nohwnd
Copy link
Member

@nohwnd nohwnd commented Aug 13, 2025

Description

This is a draft of change that we would like to make in vstest. For a long time we've been using sha1 to hash the test ids, which we want to remove for these reasons:

  • we use it for non-crypto purposes, but we cannot be explicit about it because it is a cryptographic algorithm, so someone can get the wrong idea that this is security related
  • it is slow

We would like to replace with xxhash128, and a versioned guid (v8 guid, with 4 bits allocated for our version, so we can tell the guid is new and what hashing algorithm was used - not implemented in the code yet)

Impact

  • Flaky test detection in AzDO might be impacted, but the data will replenish as new builds will run.
  • TestCase association to test plans should not be impacted.
  • Test Adapter / Test Framework authors might be impacted based on how they use TestCase.Id

MSTest

MSTest already uses xxhash128 in v4. v3 relies on object model 17.14 which does not have the change.

NUnit

Nunit relies on eqthash, or keeps the ID to the default, depending on the setting UseNUnitIdforTestCaseId which is said to maybe cause problems, so it defaults to testcaseId]

https://github.com/nunit/nunit3-vs-adapter/blob/master/src/NUnitTestAdapter/TestConverter.cs#L194-L197

https://grep.app/search?f.repo=nunit%2Fnunit3-vs-adapter&q=eqthash.guidfromstring

XUnit

Our code is not used, but SHA1 is used. Might be a good idea to move to xxhash128 as well.
https://github.com/xunit/visualstudio.xunit/blob/28c6028b64ffb700922e1a0c8d99ea041bfd03fa/src/xunit.runner.visualstudio/Sinks/VsDiscoverySink.cs#L121

BenchmarkDotNet

https://github.com/dotnet/BenchmarkDotNet/blob/e2d30d37433444b0f43088727a14885d76c56bfc/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs#L85-L93

Should move to the new testIdProvider.

Others

No other usages for eqtHash and testIdProvider were found on grep.app, so many people probably depend on the default behavior implemented in TestCase.

Opt-out

To be defined.

Environment variable and runsetting seem like the natural choice to set this per run, but that also seems quite annoying.

Assembly level attribute could be set by a test framework via build targets to opt-out.

We could also publish a way to get the default sha1 based test ID, so the adapters can set it totp TestCase.Id, and avoid the built in way to calculate the id.

https://github.com/microsoft/vstest/blob/main/src/Microsoft.TestPlatform.ObjectModel/TestCase.cs#L173

In any case the goal is to remove the SHA1 based api soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant