Inline task configs are not validated when setting a pipeline

[sokoow]

Summary

I'm trying to pass an output of a task into an input of another task, and it's does not seem to be working

Steps to reproduce

Given and ran this pipeline manifest:

---
resources:
  - name: puczatbot_source
    type: git
    source: &repo-source
      uri: XXX
      branch: master
      username: user
      password: pass

  - name: puczatbot_image
    type: registry-image
    icon: docker
    source:
      repository: XXX

  - name: golang-1.18-image
    type: registry-image
    icon: docker
    source:
      repository: golang
      tag: 1.18

jobs:
  - name: build-and-push
    plan:
      - get: puczatbot_source
        trigger: true

      - get: golang-1.18-image
        trigger: true

      - task: debug-step
        config:
          platform: linux
          image_resource:
            type: registry-image
            source: {repository: alpine}
          inputs:
          - name: puczatbot_source
          run:
           path: /bin/sh
           args:
              - -cx
              - |
                ls -al
                export
                pwd
 
      - task: build-puczatbot
        image: golang-1.18-image
        config:
          platform: linux
          inputs:
          - name: puczatbot_source
          ouputs:
          - name: puczatbot-artifacts
          run:
            path: /bin/sh
            args:
              - -cx
              - |
                cd puczatbot_source
                ls -al
                pwd
                go clean -modcache
                go get -d
                go mod download
                go build -buildvcs=false
                ls -al
                cp puczatbot ../puczatbot-artifacts

      - task: debug-step2
        config:
          platform: linux
          image_resource:
            type: registry-image
            source: {repository: alpine}
          inputs:
          - name: puczatbot-artifacts
          run:
           path: /bin/sh
           args:
              - -cx
              - |
                ls -al
                export
                pwd

      - task: build-docker-image
        privileged: true # oci-build-task must run in a privileged container
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              repository: concourse/oci-build-task
          inputs:
          - name: puczatbot_source
          - name: puczatbot-artifacts
          outputs:
          - name: image
          params:
            CONTEXT: puczatbot_source/
          run:
           path: build

      - put: puczatbot_image
        params:
          image: image/image.tar

In step build-puczatbot I'm trying to create an output named puczatbot-artifacts and then use it in steps debug-step2 and build-docker-image. I'm getting missing inputs: puczatbot-artifacts

Expected results

I'd expect output of a task to be possible to be passed as input of another task

Actual results

missing inputs: puczatbot-artifacts

Additional context

Running on kubernetes 1.29 with containerd runtime 1.7

Triaging info

• Concourse version: v7.11.2 , containerd runtime
• Did this used to work? According to documentation, yes

[sokoow]

In the logs, I see some of these:

{"timestamp":"2024-03-12T09:19:10.601106570Z","level":"error","source":"worker","message":"worker.garden.garden-server.attach.failed","data":{"error":"load proc: no running process found: process does not exist resource: not found","handle":"7b9d13e6-baef-4962-785d-d5babbec4aa3","session":"1.2.27"}}
{"timestamp":"2024-03-12T09:19:16.872933551Z","level":"error","source":"worker","message":"worker.garden.garden-server.attach.failed","data":{"error":"load proc: no running process found: process does not exist resource: not found","handle":"ae76cf1e-a9c7-4e87-6fdd-c6a1e2b40582","session":"1.2.33"}}
{"timestamp":"2024-03-12T09:20:04.832937184Z","level":"error","source":"worker","message":"worker.garden.garden-server.attach.failed","data":{"error":"load proc: no running process found: process does not exist resource: not found","handle":"6438d90f-81bb-4593-52b2-eb54c89f311c","session":"1.2.49"}}
{"timestamp":"2024-03-12T09:20:12.227481890Z","level":"error","source":"worker","message":"worker.garden.garden-server.attach.failed","data":{"error":"load proc: no running process found: process does not exist task: not found","handle":"a6eb268f-089f-4608-5940-853538eacdf3","session":"1.2.60"}}

[sokoow]

might be related to #8676

[sokoow]

also related to #7546

[taylorsilva]

I think this is a pipeline config issue...

I modified the pipeline a bit to run on my concourse:

---
resources:
  - name: puczatbot_source
    type: git
    source:
      uri: https://github.com/concourse/concourse.git
      branch: master

  - name: puczatbot_image
    type: registry-image
    icon: docker
    source:
      repository: concourse/concourse

  - name: golang-1.18-image
    type: registry-image
    icon: docker
    source:
      repository: golang
      tag: 1.18

jobs:
- name: build-and-push
  plan:
  - get: puczatbot_source
    trigger: true
  - get: golang-1.18-image
    trigger: true
  - task: debug-step
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: alpine}
      inputs:
      - name: puczatbot_source
      run:
       path: /bin/sh
       args:
          - -cx
          - |
            ls -al
            export
            pwd

  - task: build-puczatbot
    image: golang-1.18-image
    config:
      platform: linux
      inputs:
      - name: puczatbot_source
      ouputs:
      - name: puczatbot-artifacts
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            cd puczatbot_source
            ls -al
            pwd
            ls -al
            cp README.md ../puczatbot-artifacts/

  - task: debug-step2
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: alpine}
      inputs:
      - name: puczatbot-artifacts
      run:
       path: /bin/sh
       args:
          - -cx
          - |
            ls -al
            export
            pwd

  - task: build-docker-image
    privileged: true # oci-build-task must run in a privileged container
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: concourse/oci-build-task
      inputs:
      - name: puczatbot_source
      - name: puczatbot-artifacts
      outputs:
      - name: image
      params:
        CONTEXT: puczatbot_source/
      run:
       path: build

  - put: puczatbot_image
    params:
      image: image/image.tar

Initially while modifying the pipeline to get it running, I noticed that the output dir puczatbot-artifacts was not present in the task build-puczatbot:

┌─[taylor-macbook][~/workspace]
└─▪ fly -t ci i -u https://ci.concourse-ci.org/teams/main/pipelines/taytest/jobs/build-and-push/builds/1
1: build #1, step: build-puczatbot, type: task
2: build #1, step: debug-step, type: task
3: build #1, step: golang-1.18-image, type: get
4: build #1, step: image, type: check, expires in:
5: build #1, step: image, type: get
choose a container: 1
root@fa360782-25c5-4d78-5893-b5cf2a7d308d:/tmp/build/89d4ce12# ls
puczatbot_source
root@fa360782-25c5-4d78-5893-b5cf2a7d308d:/tmp/build/89d4ce12#
exit

This task doesn't fail because the cp you run copies and renames the file. You need to add a / at the end of the command like this:

- cp puczatbot ../puczatbot-artifacts
+ cp puczatbot ../puczatbot-artifacts/

So at this point I'm like "okay, there's gotta be a typo or something here" and sure enough:

      - task: build-puczatbot
        image: golang-1.18-image
        config:
          platform: linux
          inputs:
          - name: puczatbot_source
-         ouputs:
+         outputs:
          - name: puczatbot-artifacts
          run:
            path: /bin/sh

You dropped a "t" 🙃
I am disappointed that Concourse didn't yell about "ouputs" being invalid. I actually thought we checked for that stuff because I know we do check a bunch of stuff: https://github.com/concourse/concourse/blob/master/atc/configvalidate/validate.go

I guess inline task configs aren't checked though :(

If I run that one task as a one-off using fly execute:

platform: linux
image_resource:
  type: registry-image
  source:
    repository: busybox

inputs:
  - name: puczatbot_source
ouputs:
  - name: puczatbot-artifacts

run:
  path: /bin/sh
  args:
    - -cx
    - |
      cd puczatbot_source
      ls -al
      pwd
      ls -al
      cp README.md ../puczatbot-artifacts/

Result:

┌─[taylor-macbook][~/workspace]
└─▪ fly -t ci e -c task.yml -j taytest/build-and-push
error: error unmarshaling JSON: while decoding JSON: json: unknown field "ouputs"

AHH! The expected error! So the real issue here is that inline task configs are not checked by the pipeline validator.