Fully customizable pipeline definitions

[KuraiAndras]

I have encountered a number of projects where I had to customize pipelines in ways that the Nuke pipeline attributes could not handle. My usual solution would be to either manually write the pipeline or do some hacky workaround to generate a customized pipeline file.

I have been thinking and came up with a solution I would like to pursue, and I would like to get some critique before pushing forward (If I finish this then it will be open sourced). Keep in mind that I have mostly used nuke with GitHub Workflows and Azure Pipelines.

I like nuke's generated pipelines, but they have a few problems:

• No support for running on self-hosted runners
• No support for running inside containers
• Built-in steps might be behind on versions
• Sometimes the easiest and most sane solution is to have steps before you run nuke (my case was I needed to install cloudsmith)

So, I thought about making an alternative which gives you full control of the generated files.

The sample build we want to run:

partial class Build : NukeBuild
{
    public static int Main() => Execute<Build>(x => x.Run);

    [Parameter("Configuration to build - Default is 'Debug' (local) or 'Release' (server)")]
    readonly Configuration Configuration = IsLocalBuild ? Configuration.Debug : Configuration.Release;

    Target SayHello => _ => _
        .Executes(() => Log.Information("Saying Hello"));

    Target Run => _ => _
        .Executes(() => Log.Information("Run completed, configuration is {Configuration}", Configuration));
}

Current generation parameters:

[GitHubActions
(
    "ci",
    GitHubActionsImage.UbuntuLatest,
    FetchDepth = 0,
    OnPushBranches = new[] { "main" },
    CacheExcludePatterns = new[] { "*.tar" },
    InvokedTargets = new[] { nameof(Run), nameof(SayHello) }
)]

My approach:

// We specify a factory type so we are not restricted to primitives
[GithubActionsExtended(typeof(Factory))]
partial class Build
{
    // Factpry is nested type, so we don't need to specify Build.SayHello and others
    public class Factory : IGithubActionsImplementationFactory
    {
        // Using the new target-typed new expressions helps a lot but it is a bit too verbose
        // Might be improved by terse formatting?
        public GithubWorkflowDefinition Create() => new()
        {
            FileName = "extended",
            Name = "extended",
            On = new()
            {
                Push = new()
                {
                    Branches = new[]
                    {
                        "main"
                    }
                }
            },
            Jobs = new()
            {
                { "ubuntu-latest", new()
                    {
                        Name = "ubuntu-latest",
                        RunsOn = "ubuntu-latest",
                        // Fully custom steps
                        Steps = new Step[]
                        {
                            new Checkout() { FetchDepth = 0 }, // Custom checkout step
                            new Cache() { CacheExcludePatterns = new[]{ "*.tar" } }, // Custom cache step
                            new Step(){ Run = "echo \"After\"" }, // Generun step
                            new NukeRun() { InvokedTargets = new[]{ nameof(Run), nameof(SayHello) } }, // cusrom step for running nuke
                            new Step(){ Run = "echo \"Before\"" }
                        },
                    }
                }
            }
        };
    }
}

Right now, this will produce the following output:

# ------------------------------------------------------------------------------
# <auto-generated>
#
#     This code was generated.
#
#     - To turn off auto-generation set:
#
#         [GithubActionsExtended (AutoGenerate = false)]
#
#     - To trigger manual generation invoke:
#
#         nuke --generate-configuration GitHubActions --host GitHubActions
#
# </auto-generated>
# ------------------------------------------------------------------------------

name: extended
on:
  push:
    branches:
      - main
jobs:
  ubuntu-latest:
    name: ubuntu-latest
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: 'Cache: .nuke/temp, ~/.nuget/packages'
        uses: actions/cache@v3
        with:
          path: >-
            .nuke/temp

            ~/.nuget/packages

            !*.tar
          key: ${{ runner.os }}-${{ hashFiles('**/global.json', '**/*.csproj', '**/Directory.Packages.props') }}
      - run: echo "After"
      - name: 'Run: Run SayHello'
        run: ./build.cmd Run SayHello
      - run: echo "Before"

Stuff that's missing but might be possible:

• Automatically handle targets with .Produces()
• Automatically import secrets from .Requires()
• Add all missing properties from GitHub 's documentation
• Supporting other providers - Azure, Jenkins, AppVeyor, etc...

So how does this work?
The basic idea is that surely GitHub workflows can be represented with simple POCOs which are then serialized to YAML. So the GithubWorkflowDefinition created by the factory will be serialized using tools provided by nuke's ConfigurationAttributeBase

some code from the implementation:

public class GithubWorkflowDefinition
{
    [YamlIgnore]
    public required string FileName { get; init; }

    public required string Name { get; init; }
    public string? RunName { get; init; }

    public required On On { get; init; }

    public required Dictionary<string, Job> Jobs { get; init; }
}
public class Step
{
    [YamlMember(Order = int.MinValue)]
    public virtual string? Name { get; set; }
    [YamlMember(Order = int.MinValue + 1)]
    public virtual string? Uses { get; set; }
    [YamlMember(Order = int.MinValue + 2)]
    public virtual string? Run { get; set; }
    [YamlMember(Order = int.MinValue + 3)]
    public virtual Dictionary<string, object>? With { get; set; }
}
public class Checkout : Step
{
    [YamlIgnore]
    public string Version { get; init; } = "3";

    [YamlIgnore]
    public int? FetchDepth { get; init; }

    public override string? Uses => $"actions/checkout@v{Version}";

    public override Dictionary<string, object>? With => FetchDepth is null
        ? null
        : new()
        {
            { "fetch-depth", FetchDepth },
        };
}
public class NukeRun : Step
{
    [YamlIgnore]
    public required string[] InvokedTargets { get; init; }

    [YamlIgnore]
    public string RunCommand { get; init; } = "./build.cmd";

    public override string? Name => $"Run: {string.Join(' ', InvokedTargets)}";

    public override string Run => $"{RunCommand} {string.Join(' ', InvokedTargets)}";
}

Stuff that is not in scope:

• Making it 100% bulletproof. I do not plan on making this in a way which prohibits the user from making a faulty pipeline definition
• Making it terser/more beginner friendly than nuke's current implementation - if you like using the current way of generating pipeline definitions then keep using them. This would be for scenarios where more customization is needed, but you still want nuke to do some heavy lifting

I hope some of you might like this idea, so if you have some feedback, please share (even if you don't like this idea!)

[BusHero]

Why use a C# DSL to generate a pipeline instead of writing the pipeline manually?

[matkoch]

nuke already has all the information about build dependencies.. it makes perfect sense to generate the yaml from that single source of truth..

[BusHero]

It makes sense in the case of simpler pipelines, not so sure if it scales to more complex scenarios. These scenarios may involve:

• Custom steps
• Multi job pipeline
• Vendor dependent features
• Multi vendor pipelines.

Let's say nuke does end up generating pipelines for all of these exotic scenarios, how much configurability will nuke allow? Will you have the full control over the generated pipeline? If your goal is maximum configurability, you will end up with a C# DSL that resembles pipeline's YAML dialect. Does this provide more benefits than writing the pipeline directly? I'm doubtful about.

[matkoch]

I'm not doubtful. it's just a challenge that has to be accepted by someone.

[Danielku15]

I also wanted to add some bits to this dicussion. I also decided to move to Nuke for one of my new OSS projects and we likely also want to adopt it in our company for most of our pipelines written in a variety of languages (powershell, batch, msbuild,..). The post is very long but might give some good insight into what I am thinking.

tl;dr: How about, as a first improvement step, we adopt the "GitHub Actions" concept of an "Action" into Nuke? As a "unit" which has inputs, performs steps and produces outputs it can be reused and integrated into Targets. Those special "Actions" can be called multiple times, are producing special build logs (grouping, time measurement etc.) and might even be shown in the execution plans.

Long version follows here

The problem

Task/Target based runners have definitly their place and fulfill a nice range of usecases but they suffer often of similar pain points when used in larger workflows:

• Reusing tasks with multiple inputs is often not easily achievable. e.g. how would a matrix build in Nuke look like? Or in MsBuild you might use batching but multiple CallTargets also fails to serve this need.
• Having conditional branching is often a pain. You end up with tons of conditions each on individual task level do ensure that in all the different pipelines a task might be used, it is considered correctly.
• Controlling the right order of parallel/sequential execution is often tricky.

Here a screenshot of the project I work currently on. This is "only" the GitHub Actions level jobs:

In the jobs I often have a Matrix strategy calling parameterized a single target. There are also additional inputs from outside. like whether some cached files should be used or not.

The entrypoint is often a target with certain parameters which then has dependent targets being executed. It took me really long time and many tradeoffs to make certain things reusable. I have plenty of OnlyWhenStatic, DependsOn and Before mixing targets from different pipelines in the shared ones. Yes it works, but it's not really nice.

Also my pipeline is rather encoded in GitHub Actions yamls and executing the same range of builds locally is not possible out-of-the-box.

For many developers it might be a lot more natural to develop conditions, loops and call methods than working on a task list and dependencies. Yes, you can do that today, but you will loose a lot of Nuke capabilities like nice outputs, time measurement, execution plan visualization etc.

Brainstorming a solution

I haven't thought through how such a pipeline syntax in Nuke could really look like but it is anyhow more important first to find out what conceptual approach Nuke should follow. I think it could be based on various aspects of GitHub Actions:

• A new standalone building block named NukeAction defines a certain unit of work. The idea of them is to be reusable and sharable (e.g. as NuGet packages). A separate construct avoids confusion and ensures only available features are listed (e.g. actions wouldn't have .DependsOn, .After, .Before etc). They have some similarities with the Built-In CLI Tools with Fluent APIs but follow a bit a different mindset.
• NukeActions when used as "steps" should produce log outputs similar to targets with log grouping in CI systems supporting, time measurement etc. This is important just like for target sequences to see how the pipeline is progressing and executing.
• NukeActions can be called from other NukeActions, in Targets as dependency, in Targets manually in the execution function and also in "Jobs". This way they can be a benefit to classical Target approaches and newer pipeline approaches.
• NukeJobs are consisting of steps calling predefined NukeActions or inline actions. For each step there are conditionals whether to execute the step. They fulfill the role of certain "stages" in the overall pipeline.
• NukeWorkflows are consisting of NukeJobs put into order (sequential, parallel) and the execution paths might be enriched with conditions. The workflows are putting all the things together into final pipelines.

Today we have "only" Targets which are somehow a mix between Jobs and Workflows.

The concepts would compare like this:

Nuke: NukeAction

• GHA: Actions
• TeamCity: Steps
• Azure DevOps: Tasks
• Jenkins: Steps

Nuke: NukeJob

• GHA: Jobs
• TeamCity: Build Configuration
• Azure DevOps: Job
• Jenkins: Job

Nuke: NukeWorkflow:

• GHA: Workflows
• TeamCity: Maybe Sub Projects or Composite Build Configurations?
• Azure DevOps: Pipelines
• Jenkins: (Multi-branch) Pipelines

Such a structure should then also allow proper code generation where the Nuke build.exe is ultimately called with the Job name. Locally the whole workflow could also be executed.

Conclusion

CI/CD systems are focusing on very similar concepts of a "Pipelines > Jobs > Steps" hierarchy which is controlled through further conditions. Nuke has some overlaps on the Steps/Jobs areas but does not fully match into those principles with primarily Targets as first-class citizen.

Adopting various principles into Nuke could be a big benefit as devs and devops' dont need to change their mindset when using Nuke but only need to know the terminology. I personally think it would fit into the principles and goals of Nuke quite well to also adopt those aspects.

The adoptions could be done stepwise with adding NukeActions as a start to have defined and reusable building blocks. Later job/workflow additions could be made as own construct with execution plans, conditionals, matrix builds etc. With actions available people might be able to build the pipelines already in the way they like and we don't even need to add everything.

References:

• https://blog.jetbrains.com/teamcity/2019/04/configuration-as-code-part-4-extending-the-teamcity-dsl/#defining-the-pipeline-in-code
• https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsif
• https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
• https://www.jenkins.io/doc/book/pipeline/syntax/