Create SPM Build Plugin to generate GraphQL code during build (prebuild command)

[ripplexyz]

Use case

Create BuildTool Plugin Target like https://github.com/apple/swift-protobuf/blob/10295a85cdf2a50ff2125aa5903fe2e18e78e556/Package.swift#L30C16-L30C35

BuildTool Plugins can be added as dependency to other Swift Packages or Xcode Projects which then runs during build process.

Describe the solution you'd like

import PackageDescription

let package = Package(
  name: "ApolloCodegen",
  platforms: [
    .iOS(.v16),
    .macOS(.v13),
    .tvOS(.v16),
    .watchOS(.v9),
  ],
  products: [
    .plugin(name: "ApolloCodegenPlugin", targets: ["ApolloCodegenPlugin"]),
  ],
  dependencies: [
    .package(url: "https://github.com/apollographql/apollo-ios-codegen", from: "1.0.0"),
  ],
  targets: [
    .plugin(
      name: "ApolloCodegenPlugin",
      capability: .buildTool(),
      dependencies: [
        .product(name: "apollo-ios-cli", package: "apollo-ios-codegen"),
      ]
    ),
  ]
)

[calvincestari]

Hi @ripplexyz, thanks for opening this issue.

This feature request makes sense if you're upgrading directly from the legacy (0.x) versions where it was recommended to have a build step in your project that would run code generation as part of each build. We no longer make that recommendation in 1.0; it's not necessary for code generation to be run with each build because the schema/operation files do not typically change between each build. It's worth noting though that you can still achieve the same build step by using the new CLI tool.

We could build an SPM build tool plugin and offer more automation for codegen to be integrated with the build process but is there a need for it? Your use case above describes the how but not the why. Can you help us understand more about why running code generation as part of each build is useful to your workflow?

[kaybutter]

Hi, I have currently wrapped the CodeGenLib in a build tool plugin myself.
My experience with the approach is:

• Changing operation files is easier for the team. Update the file. Rebuild. Use generated code. New developers won't have to learn to use cli tools.
• If set up correctly, build tool plugins will work with incremental builds to only run code generation if schema/operation files are changed. I found that difficult to achieve with a build step that just calls the cli tool.
• It decreases noise during code reviews because the generated code is not part of the repo
• CodeGeneration is also run as part of our CI builds, so if some part of it breaks we notice it right away

[ripplexyz]

@kaybutter we also have same issues, we also prefer to generate code in pluginWorkDirectory so it reduces noise from code reviews.

Can I ask how you're able to do fetch schema in build tool plugin? As we are facing issue that network request is not allowed in buildTool plugins.

[kaybutter]

@ripplexyz I don't. Updating the schema won't work with the sandboxed extension, but it would also break the incremental builds, because the build system has no way of knowing upfront whether the remote schema has changed.
You can still update it manually through Xcode using a command plugin or an executable target.

[ripplexyz]

Example download command plugin and generate buildTool plugin -> https://github.com/ripplexyz/apollo-ios-plugin

[calvincestari]

Hi, I have currently wrapped the CodeGenLib in a build tool plugin myself. My experience with the approach is:

• Changing operation files is easier for the team. Update the file. Rebuild. Use generated code. New developers won't have to learn to use cli tools.
• If set up correctly, build tool plugins will work with incremental builds to only run code generation if schema/operation files are changed. I found that difficult to achieve with a build step that just calls the cli tool.
• It decreases noise during code reviews because the generated code is not part of the repo
• CodeGeneration is also run as part of our CI builds, so if some part of it breaks we notice it right away

@kaybutter we also have same issues, we also prefer to generate code in pluginWorkDirectory so it reduces noise from code reviews.

These all seem like team workflow-specific issues in which case you've resolved the issue correctly by creating a custom build plugin that you maintain. I'm still not clear on the benefit of a build plugin for the majority of Apollo iOS users when our stance is that it's not good practice to execute codegen on each build.

• As for setting up the build tool plugin correctly to work with incremental builds and only execute codegen when operations change; we found Xcode's input/output files to be unreliable and have not revisited whether SPM improves that reliability.
• The CLI tooling is intended to work with a configuration file which would be created once and then a simple ./apollo-ios-cli generate command is all that's needed to regenerate the models. The learning curve should be limited to crafting the configuration file.
• Keeping the generated code out of the repo to simplify code review could also be satisfied with a git ignore path.
• We have CI tests that execute code generation too and we do that through the CLI tool and the associated configuration file.

I'm going to close this issue as it's not something we're planning on supporting unfortunately. I think your custom build plugin is the the right solution here.

[kaybutter]

As for setting up the build tool plugin correctly to work with incremental builds and only execute codegen when operations change; we found Xcode's input/output files to be unreliable and have not revisited whether SPM improves that reliability.

For anyone interested in this part, you can easily use code to find all the relevant input files when building a plugin. Unfortunately, the output files are trickier to predict because of the way that apollo codegen works. To know which files apollo will create, you'd kind of have to understand the content of the schema and the operation files. The schema will provide extra types, like enums, which will be created in extra files. Operation files can contain multiple queries or fragments which will all result in dedicated files as well. This does not work well with output files that have to be known before we even run codegen.

In my project, I decided to merge all files created by codegen into a single swift file. That way I had a single predictable output file and incremental builds have been rock solid for me ever since. Though maybe someone knows of a better strategy for how to achieve this.

[calvincestari]

I'm glad you've got a reliable workflow that works for you. That's the beauty of open-source.

In our experience we found Xcode input/output files to be unreliable even with a single generated file, which was an output option in the 0.x versions of Apollo iOS. The single file also chokes Xcode when the schema/operations are at scale. You'll notice that 1.x no longer supports a single output file in order to support modular architectures and resolve the Xcode file size issues.

[harish-suthar]

@calvincestari can you provide a function that provides output files to be generated before we execute the codegen?

[calvincestari]

@harish-suthar that's not possible because it's only during compilation of the operations and schema that all the files to be generated are determined. You could naively do a quick scan of the operation files to determine what the operation and fragment file paths might be but you won't know what schema types will be generated from the operation files alone.

[harish-suthar]

@calvincestari Can we have a feature request open for this requirement?

[calvincestari]

I'll speak to the rest of the team and see if there is any creative way with minimal execution we can support this workflow.

[calvincestari]

Are the output paths needed just for incremental builds or is it a requirement of the build plugin?

[kaybutter]

Are the output paths needed just for incremental builds or is it a requirement of the build plugin?

It's both. The incremental build system uses it to determine if the plugin needs to be executed. But any files not present in the output files will also not be processed by any further build steps. So if you create a whole bunch of swift files but don't list them in the outpuf files, they won't be compiled.

[calvincestari]

The incremental build system uses it to determine if the plugin needs to be executed.

Would this not only be affected by input files, i.e.: the schema and operation files? The generated files are an output of the plugin being executed. Determining plugin execution from the files it will produce itself seems backwards.

But any files not present in the output files will also not be processed by any further build steps. So if you create a whole bunch of swift files but don't list them in the outpuf files, they won't be compiled.

Sure, this makes sense. So you'll just need a list of the paths up front but the actual file content can be done by codegen as usual - is that correct?

[kaybutter]

Would this not only be affected by input files, i.e.: the schema and operation files? The generated files are an output of the plugin being executed. Determining plugin execution from the files it will produce itself seems backwards.

I'm not 100% sure about this. But I assume if the output files get deleted the step will also be executed again. So at least the existence of the files is checked.

Sure, this makes sense. So you'll just need a list of the paths up front but the actual file content can be done by codegen as usual - is that correct?

That's correct.

[calvincestari]

I'm not 100% sure about this. But I assume if the output files get deleted the step will also be executed again. So at least the existence of the files is checked.

Hmm, this could get complicated by the pruning functionality of codegen.

[harish-suthar]

@calvincestari I am not sure how it will affect the prune functionality exactly but the requirement is simply to get final list of files that will be there at the output directory path specified using the apollo config after we execute codegen swift script.

@kaybutter am I right?

[calvincestari]

@calvincestari I am not sure how it will affect the prune functionality exactly but the requirement is simply to get final list of files that will be there at the output directory path specified using the apollo config after we execute codegen swift script.

No I meant the other way around; pruning may affect the determination of plugin execution. If a file is no longer meant to be there (pruned) then it's correct that the plugin should be executed but I wonder if there is also a case where pruning could cause execution to happen when it's not needed. Just thinking out loud here..

[kaybutter]

But if a file is no longer meant to be there it also wouldn't be part of the output files anymore. 🤷‍♂️

[calvincestari]

After speaking with the rest of the team today we've decided to re-open this issue but change the solution to a Build Plugin that would return prebuild comands.

• prebuild commands — are run before the build starts and can generate an arbitrary number of output files with names that can't be predicted before running the command
• build commands — are incorporated into the build system's dependency graph and will run at the appropriate time during the build based on the existence and timestamps of their predefined inputs and outputs

The complexity here is that because of the dynamic nature of prebuild command output they are are run on each and every build. This means the plugin/command itself needs to be able to cache and handle incremental builds instead of letting the Xcode/SPM infrastructure handle that for you. Adding caching and incremental building to the codegen engine is something we've discussed before and would be a dependency for this plugin work; I've created #3310 where we can track that.

The unfortunate reality is that neither of these are high priority work on our roadmap and neither are they small pieces of work, so they're unlikely to be worked on any time soon.