Multiple exit tests result in corrupt coverage profdata, which leads to test command failure, even with `.serialized`

[simonjbeaumont]

Description

Was really happy to find exit tests and had been adopting them over the last couple of days.

I have a project where adding just one more exit test causes it to reliably fail when SwiftPM(?) hoovers up the coverage data and calls out to llvm-profdata merge:

error: terminated(1): /usr/bin/llvm-profdata merge -sparse /pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/XCTest13106936571747098128_0.profraw '/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing13106936571747098128_0.profraw' -o /pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/default.profdata output:
    warning: /pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing13106936571747098128_0.profraw: excessively large counter value suggests corrupted profile data: 270326661043459020
    error: no profile can be merged

This is occurring for me reliably in CI and locally on the 6.2 Linux snapshot toolchain.

My hypothesis here is that the path to the profdata file has been determined at the start of day, before Swift Testing forks for the exit test, and so they're all writing into the same file. Then, when SwiftPM comes along later to run llvm-profdata merge -sparse it finds a corrupt file in some cases.

This is happening even when the suite is marked as .serialized too.

I've included a reproducer below but seeing this in a real project too: https://github.com/swift-otel/swift-otel/actions/runs/16026382859/job/45215272013?pr=242#step:5:2523

Reproduction

Attaching as a zip file for convenience, but will include the entire repro contents here too.

repro-exit-tests-coverage-corruption.zip

---

Package.swift

// swift-tools-version: 6.2
import PackageDescription

let package = Package(
    name: "repro-exit-tests-coverage-corruption",
    dependencies: [
        .package(url: "https://github.com/swift-server/async-http-client", from: "1.25.0"),
    ],

    targets: [
        .testTarget(
            name: "ReproTests",
            dependencies: [
                .product(name: "AsyncHTTPClient", package: "async-http-client"),
                .product(name: "Crypto", package: "swift-crypto"),
            ]),
    ]
)

---

Tests/ReproTests/ReproTests.swift

import AsyncHTTPClient
import Testing

@Suite(.serialized) struct Suite {
    static func helper() async throws {
        await #expect(processExitsWith: .success) {
            let client = HTTPClient()
            try await client.shutdown()
        }
    }

    @Test func test0() async throws { try await Self.helper() }
    @Test func test1() async throws { try await Self.helper() }
    @Test func test2() async throws { try await Self.helper() }
    @Test func test3() async throws { try await Self.helper() }
    @Test func test4() async throws { try await Self.helper() }
    @Test func test5() async throws { try await Self.helper() }
    @Test func test6() async throws { try await Self.helper() }
    @Test func test7() async throws { try await Self.helper() }
    @Test func test8() async throws { try await Self.helper() }
    @Test func test9() async throws { try await Self.helper() }
    @Test func testA() async throws { try await Self.helper() }
    @Test func testB() async throws { try await Self.helper() }
    @Test func testC() async throws { try await Self.helper() }
    @Test func testD() async throws { try await Self.helper() }
}

---

Repro

% swift test --enable-code-coverage
Build complete! (4.02s)
Test Suite 'All tests' started at 2025-07-03 16:09:37.012
Test Suite 'debug.xctest' started at 2025-07-03 16:09:37.012
Test Suite 'debug.xctest' passed at 2025-07-03 16:09:37.012
         Executed 0 tests, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
Test Suite 'All tests' passed at 2025-07-03 16:09:37.012
         Executed 0 tests, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test test0() started.
✔ Test test0() passed after 0.023 seconds.
◇ Test test1() started.
✔ Test test1() passed after 0.022 seconds.
◇ Test test2() started.
✔ Test test2() passed after 0.022 seconds.
◇ Test test3() started.
✔ Test test3() passed after 0.022 seconds.
◇ Test test4() started.
✔ Test test4() passed after 0.022 seconds.
◇ Test test5() started.
✔ Test test5() passed after 0.021 seconds.
◇ Test test6() started.
✔ Test test6() passed after 0.021 seconds.
◇ Test test7() started.
✔ Test test7() passed after 0.023 seconds.
◇ Test test8() started.
✔ Test test8() passed after 0.022 seconds.
◇ Test test9() started.
✔ Test test9() passed after 0.023 seconds.
◇ Test testA() started.
✔ Test testA() passed after 0.023 seconds.
◇ Test testB() started.
✔ Test testB() passed after 0.022 seconds.
◇ Test testC() started.
✔ Test testC() passed after 0.022 seconds.
◇ Test testD() started.
✔ Test testD() passed after 0.022 seconds.
✔ Suite Suite passed after 0.319 seconds.
✔ Test run with 14 tests in 1 suite passed after 0.319 seconds.
error: terminated(1): /usr/bin/llvm-profdata merge -sparse /pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/XCTest12847804116295998992_0.profraw '/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing12847804116295998992_0.profraw' -o /pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/default.profdata output:
    warning: /pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing12847804116295998992_0.profraw: excessively large counter value suggests corrupted profile data: 344059485299246640
    error: no profile can be merged

Expected behavior

• Coverage data is correctly handled with exit tests.
• swift test command exits zero since all tests pass.

Environment

↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)

% swift --version
Swift version 6.2-dev (LLVM ac66248b7c45109, Swift 92fd571d0786efe)
Target: aarch64-unknown-linux-gnu
Build config: +assertions

Additional information

No response

[grynspan]

Code coverage is an LLVM feature.

[grynspan]

Sending to the SwiftPM folks first in case there's a bug in that layer, but I suspect this needs to go to the LLVM folks.

[simonjbeaumont]

My hunch is that this is a result of Swift Testing writing to the same coverage profile file concurrently, rather than an LLVM issue and would expect that this will need some coordination between Swift PM and Swift Testing.

[stmontgomery]

Swift Testing does not actually take any actions on its own when code coverage is enabled. Instead, code coverage collection happens as a result of (a) the code-under-test being built with profiling instrumentation enabled during compilation and (b) the runner process being launched with certain runtime settings, env vars, etc. set which cause the profiling data to be collected and written somewhere.

When it comes to tests in Swift packages run via swift test --enable-code-coverage, Swift package manager is the layer which is responsible for making this all work. So yes, we need SwiftPM to investigate this issue first. That said, it is possible that Swift Testing may need to take some additional action when using features like Exit Tests which spawn a subprocess; for example, it may be necessary to disable inheritance of those code coverage-related env vars/settings when spawning subprocesses, or perhaps use a different value for the env vars so that subprocess will write coverage data to a distinct location to avoid corruption. I'm not sure what the best solution is, but wanted to provide some context since it's possible that fixes in both SwiftPM and Swift Testing may be necessary to fully address this issue.

[grynspan]

It's actually intentional/desirable that the child processes contribute to code coverage. However, there have been some subtle bugs we've seen before in how LLVM generates a path for the intermediate code coverage file for a given subprocess (i.e. it can stomp its own data sometimes.) IIRC %m in the file path pattern is supposed to prevent that, but doesn't always work.

[grynspan]

See also: #7797

[stmontgomery]

Thanks @grynspan, that's helpful context. After reading the LLVM documentation I think the solution here may (in part) involve adjusting how SwiftPM specifies the LLVM_PROFILE_FILE env var: it's currently specified as \(library)%m.profraw but since Exit Tests in Swift Testing can now cause subprocesses to be spawned, the subprocesses need to write to distinct files. There's another token %p which expands to the PID of each process, and it seems like that could help. The docs even advise that:

For a program such as the Lit testing tool which invokes other programs, it may be necessary to set LLVM_PROFILE_FILE for each invocation. The pattern strings “%p” or “%Nm” may help to avoid corruption due to concurrency.

[grynspan]

%m should be sufficient per the docs. Note that XCTest's parallel testing implementation also involves multiple processes and has shown similar issues in the past. But adding %p can't make it worse (right?) so it's worth a shot.

[simonjbeaumont]

While the use of %m does speak of some locking for the pool of files, it says that's provided by the runtime, and depending on what kind of locking they're talking about, it may not be file-locking and it may not be honoured across the process boundary. Given we don't specify the N in %Nm it is inferred to be 1 too, which will exacerbate the situation.

I concur that we should add %p since we know we'll be in another process, assuming that SwiftPM will hoover everything up in the directory at the end of the day, which it looks like it should.

[simonjbeaumont]

OK, I think this is gonna work because I have concocted a (gross) workaround.

// file: ReproTests.swift

import Testing
import struct Foundation.URL
#if canImport(Darwin)
import Darwin
#elseif canImport(Glibc)
import Glibc
#endif

@Suite(.serialized) struct Suite {
    static func updateLLVMProfilePath() {
        let key = "LLVM_PROFILE_FILE"
        let profrawExtension = "profraw"
        guard let previousValueCString = getenv(key) else { return }
        let previousValue = String(cString: previousValueCString)
        let previousPath = URL(filePath: previousValue)
        guard previousPath.pathExtension == profrawExtension else { return }
        guard !previousPath.lastPathComponent.contains("%p") else { return }
        let newPath = previousPath.deletingPathExtension().appendingPathExtension("%p").appendingPathExtension(profrawExtension)
        let newValue = newPath.path(percentEncoded: false)
        print("Replacing \(key)=\(previousValue) with \(key)=\(newValue)")
        setenv(key, newValue, 1)
    }

    @Test func testA() async {
        Self.updateLLVMProfilePath()
        await #expect(processExitsWith: .success) { Subject.a() }
    }

    @Test func testB() async {
        Self.updateLLVMProfilePath()
        await #expect(processExitsWith: .success) { Subject.b() }
    }
}

// file: Subject.swift

struct Subject {
    static func a() { _ = "a" }
    static func b() { _ = "a" }
}

---

% swift test --enable-code-coverage --filter Suite.testa
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testa() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testa() passed after 0.018 seconds.
✔ Suite Suite passed after 0.018 seconds.
✔ Test run with 1 test in 1 suite passed after 0.018 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15828.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 1,
  "percent": 50
}

---

% swift test --enable-code-coverage --filter Suite.testb
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testb() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testb() passed after 0.017 seconds.
✔ Suite Suite passed after 0.017 seconds.
✔ Test run with 1 test in 1 suite passed after 0.017 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15905.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 1,
  "percent": 50
}

---

% swift test --enable-code-coverage --filter Suite.testa --filter Suite.testb
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testa() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testa() passed after 0.016 seconds.
◇ Test testb() started.
✔ Test testb() passed after 0.015 seconds.
✔ Suite Suite passed after 0.033 seconds.
✔ Test run with 2 tests in 1 suite passed after 0.033 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15981.profraw'
'Swift Testing12847901981426048528_0.15988.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 2,
  "percent": 100
}