Schema (Stitched or Federation) Adaption Architecture & Caching

[JustusNBB]

Coming from this issue here, my topic probably better fits into this discussion forum concerning my questions to this one and further use cases, I'll elaborate a bit:

Want I want:

A few bits of information for a CMS from the infrastructure. And also explorable playgrounds for its full data schema.

What I have:

OpenAPI schemata for two Entities stitched with a little piece of new business logic (additionalResolver), and two very big OData apis, one of them being MS Graph; Multiple GraphQL Playgrounds, depending on mesh and mockup configurations.

I was also planning on using a proxy, built with NestJS, which would allow me to put another layer on top - remodeling/filtering the infrastructure to a new CMS domain schema, a gateway.

Questions

After exploring graphql-mesh's capabilities of SDK lib generation, I think it is possible to remove the proxy layer in favor of SDK lib customization (filters full schema by queries by default). Instead I can also use graphql-mesh transformations, like rename, prefix, filter, extend.

• How does this work to fix the issue with MS Graph mentioned above? Overwrite the UserManager DirectoryListing type by extending it the same way federation resolvers work? Further Testing also depends on OData: Duplicate types generated for MS Graph #1838

• Should I focus on .meshrc.yml declarations to prepare a potential GraphQL federation in the future (possibly multiple yml files with env config) or should I rather favor a code first approach like the react-weatherbit-example features? Waiting on @santino's current work to get an overview of the cached data properties. It seems to me a more code oriented approach could be more flexible instead of yaml declaration(s), but it is not well documented: hydrate instance injecting env config instead of searching for files to find config to start introspecting.

• How do I fully compensate for not having another (domain) layer of abstraction on top to put the focus back onto actually modeling a fitting data schema instead of taking what is close with some easy adaptations - development flow for any future use cases. Do you have a recommended testing approach allowing to easily detect changes in source schemas and regressions? Just units tests for the generated sdk(s)?

[JustusNBB]

My last question seems to be related to this topic: #1495 - graphql-hive sounds promising!

[santino]

Hey @JustusNBB, just a quick note to say that the introspection cache work has been completed.
I will add documentation bits hopefully during the weekend, but everything is ready in the latest versions of Mesh, make sure to update all your packages to the latest versions available.

Very briefly these are the two changes you'll need:

// package.json, add the following script
"scripts": {
  ...
  "write-introspection-cache": "graphql-mesh write-introspection-cache",
  ...
}

# .meshrc.yaml, add the following property to the root (choose your file name)
introspectionCache: './introspection-cache.json',
sources: ...

Note: you need to have @graphql-mesh/cli installed, which you probably do, given you use the SDK.
When you do npm run write-introspection-cache, that will generate the json file of your choice with the raw data coming from your sources.
Then, after that, when you start your Mesh server, this will read the data from the json file rather then query your sources to introspect their raw schema.

Try to experiment with that, I hope it helps you.

[JustusNBB]

Comparing GraphQL Mesh OData SDK generation against Microsoft's Graph npm package, I noticed that quite a few dependencies are installed even for the production version of the Mesh SDK. Can the amount of production dependencies be reduced? Treeshake out everything not needed if the introspected cached adapted schema does not need it?
Could an OData or OpenAPI handler also be converted into one interfaced as it would be already GraphQL, caching on the handler level?
It's probably wrong to call this caching as it is a build time optimization. Basically I want every possible build time optimization, as if I would write everything by hand using axios - btw can I inject axios instead of using cross-fetch? - glue code to reach data services - put axios and some env config (credentials) in, get any data back, standardized by GraphQL Mesh.
Is that possible (then noone needs microsoft's npm package anymore :P)? Else I have to use multiple sources, because a single source does not justify the overhead. Anyways need #1838 fixed first. I'll only use GraphQL Mesh for my stitched OpenAPI sources for now. I want to stitch the other two OData sources too, I'll do that manually for now and reiterate on this later.

[JustusNBB]

Not to neglect the underlying introspection cache invalidation issue:
To my experience for now APIs do rarely break their backwards compatibility, rather they add things and break their schemas to the point that they will fail GraphQL Mesh introspection and thereby the instance to boot or deploy. In this case the existing introspection cache could be compared to a fresh (lazy) introspection, generating a diff for a warning of potential incompatibilities without failing to boot.

[santino]

Replied on the Issue so we keep that up to date; but happy to continue the conversation here if Discussions feels the right format; then we can summarise on the ticket just to make it relevant for future readers.

The issue you reported to Microsoft made things clearer to me.
It feels like you need to extend your schema to achieve this in a Mesh-Transforms fashion.

As about the other points you raised

Can the amount of production dependencies be reduced

I had similar discussions when landing here the first time.
The short answer is that with the current structure of Mesh runtime dependencies cannot be reduced further, you need the handlers and the handlers need their own dependencies.

However, as I mentioned the goal for v1 is to separate schema generation from schema execution.
In that case, we should be able to produce a GraphQL schema at build time that would just need to become executable at runtime.
This setup should be able to remove a lot of runtime dependencies.
Having said all that, to me really the key point is to simplify runtime jobs; I am not concerned about the bundle size, cause after all this is running on the server and doesn't have to be download by clients.

Basically I want every possible build time optimization, as if I would write everything by hand using axios

Yes, I felt like this when starting to use Mesh; I think, as explained above, we should be able to get fairly close to that in v1 if the schema generation/execution gets separated.

btw can I inject axios instead of using cross-fetch

I was pretty sure I have seen in the code options for passing your own fetcher function.
But a quick searched revealed that this is possible on two handlers only: GraphQL and OpenAPI.
In your case that should be sufficient, you should be able to pass a customFetch to your OpenAPI handler.

the underlying introspection cache invalidation issue

My personal view on this point is that this should not be a consumer's concern.
A REST API that is designed and managed decently should issue breaking changes with major versions bump.
In this case, a consumer should explicitly migrate to the new version endpoints; which removes the potential concern of an introspected schema to suddenly break; that would simply not be acceptable as an option.

[JustusNBB]

I'd still like a nice separation between what the API produces and which part of its schema I am consuming, like the list of queries to generate for an SDK.
Could the introspectionCache also be (partially) declared within the mesh config? Like the OpenAPI handler conveniently supports, with no additional file? I want ways to compare it to track how the consumed APIs evolve, even before they are used to create the executable graphql schema variant through inbuilt and custom transformations...
I'd like to easily explore, compare and diff schemas in every stage ;)

[JustusNBB]

For the last release @ardatan has a lot refactored to make introspection results a build artifact and core feature, I think this closes this discussion!