Figuring out path forward on complicated output sources

[jmchilton]

Consider cond-wf-003.cwl

class: Workflow
cwlVersion: v1.2
inputs:
  val: int
  def:
    type: string
    default: "Direct"

steps:

  step1:
    in:
      in1: val
      a_new_var: val
    run: foo.cwl
    when: $(inputs.a_new_var > 2)
    out: [out1]

outputs:
  out1:
    type: string
    outputSource:
      - step1/out1
      - def
    pickValue: first_non_null

requirements:
  InlineJavascriptRequirement: {}
  MultipleInputFeatureRequirement: {}

Currently we tie all workflow outputs to a particular step, Galaxy doesn't really have non-step processing of workflows.

Question:

• Are there cases where the outputSource wouldn't correspond to a single input or step outside the context of pickValue? If yes, if there. a simpler test case to work off of that doesn't depend on pickValue.
• My first thought here is that we can convert this output into an expression tool with a single output (or even a no-op step of some sort with a single output). Is that something in cwl-utils (https://github.com/common-workflow-language/cwl-utils) or should it be?

[jmchilton]

I found an interesting test case that uses pickValue on the result of one step.

class: Workflow
cwlVersion: v1.2
inputs:
  data:
    type: int[]
    default: [1, 2, 3, 4, 5, 6]
  test: boolean

steps:

  step1:
    in:
      in1: data
      a_new_var: test
    run: foo.cwl
    when: $(inputs.a_new_var)
    out: [out1]
    scatter: in1

outputs:
  out1:
    type: string[]
    outputSource: step1/out1
    pickValue: all_non_null

requirements:
  ScatterFeatureRequirement: {}

Problem here is very similar, we set the outputs when we create the job - not when the jobs are complete. So we don't have an extension point to pick those values and set a workflow output. Same trick here though of adding a step that takes an input, evaluates the picking, and then setting the output would potentially work.

We could also extend the workflow output logic to encode the delayed picking in the database somehow and just pick when we need to fetch the values. That isn't super clean either.

[jmchilton]

Now that I've looked through the rest of the test cases and thought through things a bit more - I think there are 4 potential paths forward.

1. pickValues are syntactic sugar, just rewrite the workflow on import to add a step with an expression tool (implement in cwl-utils)
2. pickValues are syntactic sugar, just rewrite the workflow on import to add a step with an expression tool (implement in Galaxy)
3. Somehow implement delayed workflow outputs - the progress tracker would throw a Delay exception if the outputs aren't ready and the API would return some sort of incomplete value.
4. Add a new workflow invocation state - "finalized". We could add workflow finalizer threads - that monitor scheduled workflows and usher them into a finalized state once all the jobs are complete. This is a lot of work to just implement pickValues - but we could then implement post-workflow-actions - e-mail notifications about them being complete, a performant "cleanup non-terminal datasets" option, better job states in the API, publish a PDF copy of the invocation report to an FTP directory, etc..

3 ... seems gross. It feels like a lot of complexity that isn't buying us much beyond implementing pickValues for CWL unless we can come up with some other use cases. 4 seems like I good, noble project to improve Galaxy, but it is going to be "a thing". 1 seems... very nice the only drawback is it doesn't really translate to Galaxy Format 2 readily. 1 would seem to make more sense than 2, but 2 is worth considering in case we can come up with some cool syntax that makes sense for Galaxy and Format 2 that we could adapt the outputs to (i.e. a variant of this where you're picking inputs instead of picking outputs).

[mr-c]

I can give you option 1 today and you can switch to other options at your leisure 🙂

[mr-c]

@jmchilton I've got a WIP on option 1; question: is pickValue only a problem to you when it shows up in the outputs of a CWL Workflow, but okay if it appears as part of a step? Or is it always a problem?

[mr-c]

@jmchilton common-workflow-language/cwl-utils#43

[jmchilton]

Does it ever appear on a step? I couldn't find an example of this.

[mr-c]

@jmchilton It can https://www.commonwl.org/v1.2/Workflow.html#WorkflowStepInput ; imagine a conditional when in the "middle" of a workflow.

[mr-c]

Examples:
https://github.com/EBI-Metagenomics/pipeline-v5/blob/c9904f1c6c3994598d7044361aa065736fbfdeac/workflows/raw-reads-wf--v.5-cond.cwl
https://github.com/EBI-Metagenomics/pipeline-v5/blob/c9904f1c6c3994598d7044361aa065736fbfdeac/workflows/conditionals/amplicon/amplicon-1.cwl
https://github.com/EBI-Metagenomics/pipeline-v5/blob/c9904f1c6c3994598d7044361aa065736fbfdeac/workflows/amplicon-wf--v.5-cond.cwl
https://github.com/EBI-Metagenomics/pipeline-v5/blob/c9904f1c6c3994598d7044361aa065736fbfdeac/workflows/conditionals/raw-reads/raw-reads-1.cwl
https://github.com/EBI-Metagenomics/pipeline-v5/blob/dd0180fc03e41724a1a227623e0f20f984eaca18/workflows/assembly-wf--v.5-cond.cwl
https://github.com/EBI-Metagenomics/pipeline-v5/blob/54459b5dddebaaeb27238f8bbedc7cfa77c03fb8/workflows/subworkflows/cmsearch-condition.cwl
https://github.com/EBI-Metagenomics/pipeline-v5/blob/c9904f1c6c3994598d7044361aa065736fbfdeac/workflows/subworkflows/seqprep-subwf.cwl

All of these are cwl-v1.2.0-devN; so apply s/only_non_null/the_only_non_null/g to the pickValue fields as you read them. (cwltool can handle them with --enable-dev)