Template Experiment

This page describes an experiment conducted by moritz that introduced templating to generated JSON for the GoCD JSON configuration plugin.

A similar setup was created for the GRiSP project, using Mustache templating, and Erlang for pipeline creation by nextl00p. See the generation logic, the generated pipelines and the build scripts.

Setup

There are two git repositories:

• go-pipelines-source contains templates written in the jsonnet data template language, and jsonnet files. It also contains a git repository that calls jsonnet on each source file to generate full JSON for GoCD pipelines
• go-pipelines-generated contains only generated files. It is filled by the shell script from go-pipelines-source. This repository is used as a configuration repository in GoCD.

In addition, there is a GoCD pipeline called gocd-pipelines that checks out go-pipelines-source, runs the generation script, and pushes the result to go-pipelines-generated.

Thus, the workflow for creating a new pipeline is:

• Add a source file to go-pipelines-source, git add, git commit, git push.
• The pipeline gocd-pipelines runs, creates the full JSON <yourpipeline>.gopipeline.json files in the go-pipelines-generated repository, and commits and pushes the generated files
• GoCD polls the config repository go-pipelines-generated and learns about the new pipeline.

Context

The vast majority of our pipelines follow the same scheme:

• Build a package (Debian or Python)
• upload to a mirror for test
• (optionally) deploy to a test environment
• (optionally) run end-to-end tests
• manual approval
• upload to a mirror for production
• (optionally) deploy to production

but there is a lot of variability in the details: some packages need to be built on just one Debian distribution, others on multiple. Some are deployed, some are just available to be drawn as dependencies. Some have e2e/system tests in the test environment, some don't. Some Python packages aren't actually moved to production, because they are included in other Debian packages (fat-packaging using dh-virtualenv).

Results

We started with roughly 140 pipelines, 120 of which used GoCD's built-in pipelines.

The 120 templated pipelines were all replaced by pipelines generated from jsonnet templates.

This reduced the number of templates from 12 to 3, and reduced the number of lines of GoCD configuration by 53%. In addition, we managed to reduce the number of stages by 96, which previously were only filled with placeholders, because GoCD templates don't allow conditional inclusion of stages.

Analysis

These were the features that allowed me to reduce both the number of templates and lines of code generally:

• being able to template everything, including resources (and not just as string, as arrays)
• conditionals that allowed me to exclude tasks or even stages where not necessary, thus avoiding either dummy tasks or duplication
• the ability to use (computed) defaults meant less repetition when instantiating the templates
• with list comprehensions I was able to pass in a list of Debian distributions to build packages on, and have the template generate one job per distribution, and later in the pipeline one job per distribution that d uploads each package to the corresponding repository.

Conclusion

Having a fully-fledged template system between the hand-edited files and what GoCD consumes adds a lot of flexibility that GoCD's built-in template does not provide, but that is very beneficial for real-world uses cases.