Trouble with generated ADO pipeline

[bensmooth]

I'm attempting to use NUKE to build my library in an auto-generated ADO pipeline.

I made a repo to demonstrate what I'm talking about:
https://github.com/bensmooth/NukeAdoPipelineTest/tree/master
Note the logging of files in bin\ in the Compile and Pack targets in my Build.cs.

This is the content of the auto-generated azure-pipelines.CI.yml:

# ------------------------------------------------------------------------------
# <auto-generated>
#
#     This code was generated.
#
#     - To turn off auto-generation set:
#
#         [AzurePipelines (AutoGenerate = false)]
#
#     - To trigger manual generation invoke:
#
#         nuke --generate-configuration AzurePipelines_CI --host AzurePipelines
#
# </auto-generated>
# ------------------------------------------------------------------------------

stages:
  - stage: windows_latest
    displayName: 'windows-latest'
    dependsOn: [  ]
    pool:
      vmImage: 'windows-latest'
    jobs:
      - job: Restore
        displayName: 'Restore'
        dependsOn: [  ]
        steps:
          - checkout: self
            lfs: true
          - task: Cache@2
            displayName: 'Cache: nuke-temp'
            inputs:
              key: $(Agent.OS) | nuke-temp | **/global.json, **/*.csproj, **/Directory.Packages.props
              restoreKeys: $(Agent.OS) | nuke-temp
              path: .nuke/temp
          - task: Cache@2
            displayName: 'Cache: nuget-packages'
            inputs:
              key: $(Agent.OS) | nuget-packages | **/global.json, **/*.csproj, **/Directory.Packages.props
              restoreKeys: $(Agent.OS) | nuget-packages
              path: $(USERPROFILE)/.nuget/packages
          - task: CmdLine@2
            displayName: 'Run: Restore'
            inputs:
              script: './build.cmd Restore --skip'
      - job: Compile
        displayName: 'Compile'
        dependsOn: [ Restore ]
        steps:
          - checkout: self
            lfs: true
          - task: Cache@2
            displayName: 'Cache: nuke-temp'
            inputs:
              key: $(Agent.OS) | nuke-temp | **/global.json, **/*.csproj, **/Directory.Packages.props
              restoreKeys: $(Agent.OS) | nuke-temp
              path: .nuke/temp
          - task: Cache@2
            displayName: 'Cache: nuget-packages'
            inputs:
              key: $(Agent.OS) | nuget-packages | **/global.json, **/*.csproj, **/Directory.Packages.props
              restoreKeys: $(Agent.OS) | nuget-packages
              path: $(USERPROFILE)/.nuget/packages
          - task: CmdLine@2
            displayName: 'Run: Compile'
            inputs:
              script: './build.cmd Compile --skip'
      - job: Pack
        displayName: 'Pack'
        dependsOn: [ Compile ]
        steps:
          - checkout: self
            lfs: true
          - task: Cache@2
            displayName: 'Cache: nuke-temp'
            inputs:
              key: $(Agent.OS) | nuke-temp | **/global.json, **/*.csproj, **/Directory.Packages.props
              restoreKeys: $(Agent.OS) | nuke-temp
              path: .nuke/temp
          - task: Cache@2
            displayName: 'Cache: nuget-packages'
            inputs:
              key: $(Agent.OS) | nuget-packages | **/global.json, **/*.csproj, **/Directory.Packages.props
              restoreKeys: $(Agent.OS) | nuget-packages
              path: $(USERPROFILE)/.nuget/packages
          - task: CmdLine@2
            displayName: 'Run: Pack'
            inputs:
              script: './build.cmd Pack --skip'
          - task: PublishBuildArtifacts@1
            displayName: 'Publish: nuget'
            inputs:
              artifactName: nuget
              pathToPublish: 'bin/nuget'
      - job: CI
        displayName: 'CI'
        dependsOn: [ Compile, Pack ]
        steps:
          - checkout: self
            lfs: true
          - task: Cache@2
            displayName: 'Cache: nuke-temp'
            inputs:
              key: $(Agent.OS) | nuke-temp | **/global.json, **/*.csproj, **/Directory.Packages.props
              restoreKeys: $(Agent.OS) | nuke-temp
              path: .nuke/temp
          - task: Cache@2
            displayName: 'Cache: nuget-packages'
            inputs:
              key: $(Agent.OS) | nuget-packages | **/global.json, **/*.csproj, **/Directory.Packages.props
              restoreKeys: $(Agent.OS) | nuget-packages
              path: $(USERPROFILE)/.nuget/packages
          - task: CmdLine@2
            displayName: 'Run: CI'
            inputs:
              script: './build.cmd CI --skip'

I am using Microsoft-provided agents, and am not self-hosting.

The problem is that each target gets put into its own job, which means that the changes to the filesystem in previous steps don't carry over into the next steps. The build fails on the Pack step for this reason, because I have SetNoBuild(true) in Pack, and the bin/ output directory is empty.

Output after Compile (lots of built files):

The output from ADO in the Pack target lists no files.

Am I missing something fundamental, or is this intended behavior? I realize I could just roll my own YML that just invokes the CI target, but I was hoping to leave the YML to NUKE, since it should be the main source of truth for how to build the library.

[bensmooth]

So the TL;DR is: why does AzurePipelinesAttribute generate pipeline YML with each target in a separate job? How can the output of one target be fed into the next (Compile generates binaries, Pack sticks the existing binaries into a package) with the auto-generated YML?

[bensmooth]

I was able to work around this behavior by adding this to my AzurePipelines attribute:

NonEntryTargets = new[] { nameof(Restore), nameof(Compile), nameof(Pack) },

See NonEntryTargetsWorkaround

This skips creating separate jobs for all of the dependent targets - which creates much saner pipeline YML. Each target runs serially in the same job instead of in parallel jobs.

It is still mysterious to me why the default behavior generates parallel jobs for targets that are ran serially locally. Additionally, it is a bit of a bummer that neither NonEntryTargets nor ExcludedTargets have XML documentation, especially since it is non-obvious what either of them do.