Support swift test --id TestSuite.testName --id OtherTestSuite.testName

[weissi]

Description

There are use cases where another harness needs to or wants to invoke very specific tests in individual processes. Examples that come to mind:

• Running specific tests first (maybe some tests that have historically failed more often)
• Gathering code coverage in parallel but individually per test case (setting LLVM_PROFILE_FILE to a unique path per test)
• ...

For these use cases (and I guess others) it would be beneficial to not select the test case via regex but via its ID (as obtained through swift test list).

Furthermore, contrary to swift test --filter DOES_NOT_EXIST which will succeed (just selects 0 tests), swift test --id DOES_NOT_EXIST should fail.

Expected behavior

I'd expect swift test --id TestSuite.test1 --id OtherTestSuite.test2 --id TestSuite.test3 to

• run exactly those three tests
• fail if either of them doesn't exist (or fails)
• run them in this order

Additionally it'd be great (but that's a stretch goal to work around command line length limits) if we could swift test --ids-from-stdin which would allow for example

swift test list | sort -R | swift test --ids-from-stdin

Actual behavior

There is no such functionality today, to run exactly one specific test I need to:

1. encode the test name as regex (so TestProduct.TestSuite/testFoo(bar:) becomes ^TestProduct\.TestSuite\/testFoo\(bar:\)$)
2. hope that the test actually exists (or grep for a fixed string like Executed 1 test, with 0 failures in the output)

Steps to reproduce

n/a

Swift Package Manager version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[grynspan]

Let's consider swift test --ids-from-stdin a stretch goal here; I think you can accomplish the same sort of thing with e.g. xargs on POSIX systems.

[grynspan]

@MaxDesiatov @bnbarham What do you think of this swift test enhancement?

[neonichu]

I'd say using xargs is quite different, because it'll invoke swift test multiple times, hence no unified report across all the selected tests.

[rauhul]

I'd say using xargs is quite different, because it'll invoke swift test multiple times, hence no unified report across all the selected tests.

xargs can run the command once with all the args unless you choose a count with -n iirc

[weissi]

I'd say using xargs is quite different, because it'll invoke swift test multiple times, hence no unified report across all the selected tests.

xargs can run the command once with all the args unless you choose a count with -n iirc

That's true but then we have the command line length restrictions back in place... So in the fullness of time, we'll need the --ids-from-stdin but if that's hard maybe not in v1.