Experiment with Quick testing library

[bartekpacia]

Intro

With this PR, I'm aiming to verify whether the -only-testing flag of the xcodebuild test command works with the Quick testing library. The Quick testing library works by using XCTestCase.testInvocations API to generate test methods dynamically at runtime.

Patrol does the same thing - it also uses XCTestCase.testInvocations to generate test methods dynamically at runtime.

Result

Success

Meaning: it is possible to select a single dynamically generated test (written in Quick) using xcodebuild test's -only-testing option.

Check if yourself

Clone this repo, checkout this branch, then build:

xcodebuild build-for-testing \
  -scheme Landmarks \
  -configuration Debug \
  -sdk iphonesimulator \
  -destination 'generic/platform=iOS Simulator' \
  -derivedDataPath build/ios_integ \
  -quiet

and run:

xcodebuild test-without-building \
  -xctestrun build/ios_integ/Build/Products/Landmarks_TestPlan_iphonesimulator17.0-arm64-x86_64.xctestrun \
  -only-testing 'LandmarksUITests/QuickLandmarksUITests/testA' \
  -destination 'platform=iOS Simulator,name=iPhone 15' \
  -resultBundlePath "build/ios_results_$(date +%s).xcresult"

Notice that a single test was executed.

[bartekpacia]

There's a problem with commas though, but it shouldn't be a problem for us in Patrol.

[bartekpacia]

Investigation

Call stack without -only-testing:

2023-11-22 15:40:58.596220+0100 LandmarksUITests-Runner[83742:2095202] DBG: Called testInvocations
2023-11-22 15:40:58.607792+0100 LandmarksUITests-Runner[83305:2090813] DBG: (
	0   LandmarksUITests                    0x000000010300c368 +[ParametrizedLandmarksUITests testInvocations] + 80
	1   XCTestCore                          0x00000001038ed4dc +[XCTestCase _testInvocationDescriptors] + 72
	2   XCTestCore                          0x00000001038d0900 +[XCTestSuite testSuiteForTestCaseClass:] + 276
	3   LandmarksUITests                    0x000000010300c244 +[ParametrizedLandmarksUITests defaultTestSuite] + 140
	4   XCTestCore                          0x00000001038d3604 +[XCTestSuite prepareContainerSuite:toRunTestIdentifiers:] + 376
	5   XCTestCore                          0x00000001038d33e4 +[XCTestSuite testClassSuitesForTestIdentifiers:skippingTestIdentifiers:randomNumberGenerator:] + 172
	6   XCTestCore                          0x00000001038d2dd0 -[XCTestSuite _initWithTestConfiguration:] + 752
	7   XCTestCore                          0x00000001038d3308 +[XCTestSuite testSuiteForTestConfiguration:] + 44
	8   XCTestCore                          0x000000010389df1c __53-[XCTTestRunSession initWithTestConfiguration:error:]_block_invoke_2 + 64

Call stack with -only-testing:

Testing started
2023-11-22 15:43:23.251295+0100 LandmarksUITests-Runner[83742:2095202] [Default] Running tests...
2023-11-22 15:43:23.596220+0100 LandmarksUITests-Runner[83742:2095202] DBG: Called instancesRespondToSelector with arg: example_test
2023-11-22 15:43:23.599099+0100 LandmarksUITests-Runner[83742:2095202] DBG: (
	0   LandmarksUITests                    0x000000010b0842b8 +[ParametrizedLandmarksUITests instancesRespondToSelector:] + 104
	1   XCTestCore                          0x0000000105598a00 +[XCTestCase _resolvedTestMethodSelectorForSelector:] + 32
	2   XCTestCore                          0x000000010559880c -[XCTestCase initWithSelector:] + 40
	3   XCTestCore                          0x0000000105598b04 +[XCTestCase testCaseWithSelector:] + 28
	4   XCTestCore                          0x0000000105598b40 +[XCTestCase testCaseWithSelector:testRunConfiguration:] + 48
	5   XCTestCore                          0x0000000105580558 +[XCTestSuite testSuiteForTestWithIdentifier:inTestClass:variations:] + 296
	6   XCTestCore                          0x00000001055837fc +[XCTestSuite prepareContainerSuite:toRunTestIdentifiers:] + 880
	7   XCTestCore                          0x00000001055833e4 +[XCTestSuite testClassSuitesForTestIdentifiers:skippingTestIdentifiers:randomNumberGenerator:] + 172
	8   XCTestCore                          0x0000000105582dd0 -[XCTestSuite _initWithTestConfiguration:] + 752
	9   XCTestCore                          0x0000000105583308 +[XCTestSuite testSuiteForTestConfiguration:] + 44
	10  XCTestCore                          0x000000010554df1c __53-[XCTTestRunSession initWithTestConfiguration:error:]_block_invoke_2 + 64

[bartekpacia]

This experiment has been completed successfully.

See also:

• https://stackoverflow.com/questions/77494930/does-xctest-methods-generated-dynamically-by-testinvocations-work-with-xcodebuil