Test Framework Proposal

[smyrick]

What are the key benefits we would want

[duckki]

I'm proposing a composite schema test format where each composition test contains multiple subgraph schemas in a single file. So that each test file is a self-contained test. That's easier to maintain compared to a directory with multiple schema files.

Format

The layout of test file is as following:

<header -- common for all subgraphs>

#!subgraph [<name>]

<GraphQL schema>

#!subgraph [<name>]

<GraphQL schema>

...

The #!subgraph is an annotation indicating the beginning of a new subgraph schema. The subgraph <name> is optional. If omitted, some name is generated automatically. My current implementation generates names of the form L<line-number>.

The lines come before the first #!subgraph is considered a common header for all subgraphs in the test file.

I've been using this format for composition (and query planning) testing personally for a year or so.

Example 1

An example of successful composition with a hint:

extend schema
  @link(
    url: "https://specs.apollo.dev/federation/v2.9"
    import: ["@key" "@shareable"]
  )

#------------------------------------------------------------------------------------------------
#!subgraph A

type T @key(fields: "id") {
    id: ID!
}

type S @key(fields: "id") {
    id: ID!
    data_s: String!
}

union U = T | S  #!hint: INCONSISTENT_UNION_MEMBER

type P {
    shared: U! @shareable
}

#------------------------------------------------------------------------------------------------
#!subgraph B

type T @key(fields: "id") {
    id: ID!
    data_t: String!
}

union U = T

type P {
    shared: U! @shareable
}

type Query {
    start: P!
}

This is a composition that produces a "hint" INCONSISTENT_UNION_MEMBER, which warns about inconsistent union type definitions across subgraphs. This is not an outright error in order to allow staggered/incremental subgraph updating.

This is warned since some selection paths may become infeasible. For example,

{
    start {
        shared {
            ... on S {
                data_s # not fetchable
            }
        }
    }
}

By the way, the horizontal dividers #---- ... ---- are merely cosmetic and added for readability. They have no semantics.

Example 2

This example shows a composition error case.

extend schema
  @link(
    url: "https://specs.apollo.dev/federation/v2.9"
    import: ["@key" "@shareable"]
  )

#------------------------------------------------------------------------------------------------
#!subgraph A

interface I @key(fields: "id") {
    id: ID!
}

type T implements I @key(fields: "id") {
    id: ID!
    data_t: String!
}

type S implements I @key(fields: "id") {
    id: ID!
    data_s: String!
}

type Query {
    start: I!
}

#------------------------------------------------------------------------------------------------
#!subgraph B

interface I @key(fields: "id") {  #!error: INTERFACE_KEY_MISSING_IMPLEMENTATION_TYPE
    id: ID!
}

type T implements I @key(fields: "id") {
    id: ID!
    data_t2: String!
}

The inline annotations like #!error: XXX and #!hint: YYY are not just for documentation, but also allow the testing tool to check if the implementation produced the expected errors or hints.

[smyrick]

Notes from the meeting on July 31, 2025.

• This is good to help show composition specific, but interactions across schemas is a bit harder.
  • This isn't defining the entire query plan but as an example how data would be returned for a given query
• How is this supported across languages and implementations? Does every implementation need to build a parser?