Create new structure for the user guide

[kinow]

Part of common-workflow-language/common-workflow-language#219 , for post common-workflow-language/common-workflow-language#220

Supersedes common-workflow-language/common-workflow-language#5 (I've gone through its items and added them here).

Our user guide currently follows a Software Carpentry model, that is not ideal for a wider audience. This issue contains an initial brain dump for a new structure, but it will be refined after feedback in meetings and here on GitHub 👍

Table of Contents

1. Quick start

• An under 10 minutes intro to CWL, for users who may have no idea what are CWL, cwltool, workflows, etc.
• We can use text from the episodes 1 and 2 from the current User Guide - https://www.commonwl.org/user_guide/01-introduction/index.html & https://www.commonwl.org/user_guide/02-1st-example/index.html
• Mention that we can write CommandLineTool or Workflow as entry point (not sure if other Processes are allowed too? But I think we should focus on cwltool in this user guide). I remember this was confusing for me at first.
• Explain what is a Job Order file. I remember not having a clue when I saw that text the first time.
• Shorter version of https://www.synapse.org/#!Synapse:syn2813589/wiki/401467 ?
• Point that we use JSON and YAML, maybe add a link to docs to clarify some of the syntax gotchas, like - - Tutorial inconcistencies common-workflow-language#318
  • Maybe add the existing episode YAML from the current User Guide - https://www.commonwl.org/user_guide/yaml/index.html

---

2. Prerequisites

• Install cwltool - Highlight major features of cwltool #166
• Install Docker
• Clarify local vs container execution - Clarity in "Running Tools Inside Docker" #119
• Use cwl-runner, explain about it, and the virtualenv's - be explicit about using a new virtualenv with the reference runner #64 & simple explanation in https://www.synapse.org/#!Synapse:syn2813589/wiki/401462 too

---

3. Basic Concepts

• What is CWL, what is an implementation (show or point to a list of implementations), workflow, tool, steps

---

4. Core Concepts
   4.1. CommandLineTool
   • Spaces in commands Add discussion about how command lines work; spaces in options #39
   • Arguments (tell the reader the different use cases for arguments and inputs, tell them there is a section about inputs)
     4.2. Expressions (maybe not before other concepts?)
   • External libraries and expressionLib - Advanced CWL expression JavaScript, using external libraries and expressionLib #126
     4.3. Inputs
   • We can use the episode 3 from the current User Guide - https://www.commonwl.org/user_guide/03-input/index.html
   • Explain its fields, such as default, valueFrom, etc - better document the 'default' and 'valueFrom' fields on WorkflowStepInput common-workflow-language#359
   • The episode 9 covers Array Inputs - https://www.commonwl.org/user_guide/09-array-inputs/index.html
   • The episode 11 covers Advanced Inputs (records and type unions) - https://www.commonwl.org/user_guide/11-records/index.html
   • Exclusive parameters Example for using expressions with exclusive parameters #162
   • Optional Inputs work around for "self" based input bindings for optional inputs #44
   • Several ways of defining inputs/arguments to tools and workflows - intermediate lesson: workflow design choices and the 7 levels of configuration #33
   • The episode 8 of the current User Guide covers it - https://www.commonwl.org/user_guide/08-arguments/index.html
   • Using an input output in another input - advanced lesson: adapt the type of an output for a WorkflowStep input #90
   • How to use linkMerge - advanced: extended discussion about linkMerge #117 (or maybe move to Advanced?)
   • Secondary files - secondaryFiles are undocumented common-workflow-language#270
     4.4. Outputs
   • The episodes 4, 5, and 6 (?) of the current User Guide covers it - https://www.commonwl.org/user_guide/04-output/index.html & https://www.commonwl.org/user_guide/05-stdout/index.html & https://www.commonwl.org/user_guide/06-params/index.html
   • The episode 10 of the current User Guide covers Array Outputs - https://www.commonwl.org/user_guide/10-array-outputs/index.html
   • Creating files at runtime - build on lesson 14 "Creating Files at Runtime", but for directories #134
     4.5. Records - rework Record inputs, dependent and mutually exclusive parameters #386
   • Document corner cases & workarounds - document how to reuse custom record types (what to do about inputBindings?) common-workflow-language#241
     4.6. Enums - more about enums #155
   • Maybe show an example with booleans, pros and cons - document how to model a one-or-the-other parameter #89
     4.7. Steps
     4.8. Operations - Examples of Operations #218
     4.9. ExpressionTool
     4.10. Workflows
   • The current User Guide covers it in the episode 21 - https://www.commonwl.org/user_guide/21-1st-workflow/index.html
   • Scatter
     • ScatterMethod Explain ScatterMethod #29
     • Also in the episode 23 of the current User Guide - https://www.commonwl.org/user_guide/23-scatter-workflow/index.html
   • Subworkflows/nested workflows
     • Covered in the episode 22 from the current User Guide - https://www.commonwl.org/user_guide/22-nested-workflows/index.html
   • Conditionals todos for conditional workflow documentation #191 & add CWL v1.1, v1.2 features #188
     • Also in the episode 24 of the current User Guide - https://www.commonwl.org/user_guide/24_conditional-workflow/index.html
       4.11. Environment Variables
   • Covered by the episode 12 of the current User Guide - https://www.commonwl.org/user_guide/12-env/index.html
     4.12. Requirements and Hints
   • InlineJavascriptRequirement
     • Maybe list cases where it is not really necessary? e.g. document in user guide that multiple parameter references and string concatenation are possible without InlineJavascript common-workflow-language#389
     • Covered by the episode 13 of the current User Guide - https://www.commonwl.org/user_guide/13-expressions/index.html
   • InitialWorkDirRequirement
     • Staging Files (a common topic, and appears in questions on discourse/element... maybe it deserves a more prominent location?)
       • Writable inputs - advanced file staging: writable inputs #36
       • Creating files at runtime, from episode 14 of the current User Guide - https://www.commonwl.org/user_guide/14-runtime/index.html
       • There is a whole episode 15 in the current User Guide about it - https://www.commonwl.org/user_guide/15-staging/index.html
   • DockerRequirement
     • Covered in the episode 7 of the current User Guide - https://www.commonwl.org/user_guide/07-containers/index.html
   • SchemaDefRequirement and input schemas
     • The episode 19 from the current User Guide covers it - https://www.commonwl.org/user_guide/19-custom-types/index.html
   • ShellCommandRequirement - Explain ShellCommandRequirement #159
   • SoftwareRequirement - From the 20 from the current User Guide - episode https://www.commonwl.org/user_guide/20-software-requirements/index.html
   • MultipleInputFeatureRequirement
     • Also in episode 24 of the current User Guide - https://www.commonwl.org/user_guide/24_conditional-workflow/index.html
   • SubworkflowFeatureRequirement
     • Also in episode 22 of the current User Guide - https://www.commonwl.org/user_guide/22-nested-workflows/index.html
   • ScatterFeatureRequirement
     • Also in episode 23 of the current User Guide - https://www.commonwl.org/user_guide/23-scatter-workflow/index.html
       4.13. Advanced Topics
   • File Formats. Move the episode 16 about File Formats from the current User Guide here? - https://www.commonwl.org/user_guide/16-file-formats/index.html
   • Metadata and Authors - from the episode 17 from the current User Guide - https://www.commonwl.org/user_guide/17-metadata/index.html
   • Testing and Debugging CWL Workflows
   • Parallelism (or another title, showing scatter, independent steps, scatter+pickValue, conditionals, etc) - proper lesson on parallelism #104
   • Data creation / manipulation at runtime (for example at specific mounth paths)
   • Consume & Produce CWL objects - advanced lesson: teaching scripts how to consume & produce CWL objects directly #32
   • Using containers - https://www.synapse.org/#!Synapse:syn2813589/wiki/401462 & clarify on a common question about using non-docker tools, like singularity
     • Features offered by other runners (or runner-specific features)
     • Using TypeScript for JS Expressions - https://github.com/umccr/cwl-ica/wiki/TypeScript
       4.14. Best Practices
   • Writing CWL workflows (include existing docs from https://github.com/common-workflow-library/cwl-patterns/blob/main/README.md)
   • FAIR best practices with CWL
   • Docker best practices with CWL - capture docker+CWL best practices from Denis Yuen et al. common-workflow-language#347
   • Add the episode Recommended Practices from the current User Guide here? - https://www.commonwl.org/user_guide/rec-practices/index.html

---

18. How-Tos

• Use VSCode for writing CWL workflows - https://github.com/arvados/arvados-vscode-cwl-training

---

19. Tutorials

• A listing of existing tutorials, in the CWL website, or other repositories/sites
• Develop an Introductory CWL tutorial #160
• Running CWL in Production (on Linux, HPC, with containers, k8s, etc? We can link existing docs)
• rnaseq with CWL - https://github.com/arvados/rnaseq-cwl-training (maybe we could have sections, like bioinformatics+cwl, or use tags, etc?)

---

20. FAQ

• Create a FAQ #6
• Maybe adapt some of these (or move to a workaround?) https://www.synapse.org/#!Synapse:syn2813589/wiki/401464
• A common question is whether users/devs can use singularity or other containers
• Mutually exclusive parameters? arvbox: Pipeline fails with pip/python errors? common-workflow-language#229

---

21. Extras

• Conformance Tests
• Add the episode Miscellaneous from the current User Guide here? - https://www.commonwl.org/user_guide/misc/index.html

Notes, and remember-to's

• Have the logo, and meaning of CWL. Also links to the main website, and to the specification. We must be clear what version of the specification this user guide is talking about (1.2 at the time of writing this)
• Include links to the specification, so users can read the formal definition
• Port as much of what we have as possible, avoiding creating new text unnecessarily
• Use arguments instead of inputBinding when possible - switch from inputBinding to arguments: based syntax #34
• Use cwl-runner - be explicit about using a new virtualenv with the reference runner #64
• Leave docker for later in the docs, start focusing on cwl/cwltool - Re-ordering and splitting the User Guide #141
• How to reference local scripts - write up how to reference a local script #158
• Multiple tools writing to output.txt, or how to rename it? - User Guide: Capturing Standard Output should include example of renaming output file #161
  pulling out a single item from an array - pulling out a specific member of an array #184
• Use small containers, and try to use an up to date tag - "Custom Types" & "Specifying Software Requirements" should use a smaller tool #63
• i18n - multi-lingual CWL guide #91
• Explain CWL's objects and general terms, clarify on how JS is used to manipulate inputs/outputs/objects - List of user guide issues #5 (comment)
• Bloom's taxonomy? - Learning objectives should use Bloom's Taxonomy #150 & https://en.wikipedia.org/wiki/Bloom%27s_taxonomy#/media/File:Bloom%E2%80%99s_Taxonomy_Verbs.png

Questions

• Should we document cwltool specific behavior too? cwltool:InplaceUpdateRequirement advanced lesson: inplace editing of existing files  #49

Extras

• Either document the process for a new conformance test, or link to the cwl-v1.2 or latest repository's document - advanced lesson: turning a bug report into a CWL conformance test #115

Feel free to send any suggestions, critics, ideas, etc., update this issue, or link pull requests and issues to this one or to the parent placeholder (#219).

-Bruno

[kinow]

Some useful links (references)

Guides and Reference material

• Learn Nextflow | Seqera Labs (thanks Sarah!)
• MDN - JavaScript Reference
• Vue.js Guide

Technical Writing

• Talk by former DigitalOcean technical writer — Talk - Mason Egger: Write Docs Devs Love: Ten Tips To Level Up Your Tech Writing (Video)
  • I think the author does a good job talking about gender inclusive language, internationalization, text organization, being concise, avoiding things that distract users, etc.
• Writing effective documentation | Beth Aitman | #LeadDevBerlin (Video)

[mr-c]

Questions

* Should we document cwltool specific behavior too? common-workflow-language/common-workflow-language#49 

In general, no. That particular feature is now part of the CWL standards without a prefix. It should be documented with a "v1.1+" badge.

https://www.commonwl.org/v1.1/CommandLineTool.html#InplaceUpdateRequirement

By now, everyone supports CWL v1.2 , so the user guide should use that version and not spend much time, if any, about differences from the previous versions.

[swzCuroverse]

Adding this in case you don't have it (II might have missed it)- we should explain what an implementation is - and explain the difference between cwl-tool and other implementations so it is clear there are other implementations if you want to run at scale/have an platform/want commercial support/want to integrate with another tool (i.e. Galaxy)

I have another list Peter/I drew up awhile ago so I will find it and cut and paste here in a bit.

[swzCuroverse]

Additional thoughts - I think at the beginning you should explain the general difference between a tool, workflow and expression tool. I think that is what you have in the Basics but just in case I am putting this here:

Other topics that might be of interest include:

• FAIR best practices with CWL
• Testing and Debugging CWL Workflows
• Integration with existing Workflow and Tool Repositories
• Guides on Running CWL in Production on Specific Platforms
• • These could just be links to existing documentation to the fully supported platforms listed on the website
• We were hoping to come up with templates or a library of common used actions but some of these could be explained in the user guide
  ** https://github.com/common-workflow-library/cwl-patterns/blob/main/README.md

You can use Peter's tutorial here for some examples: https://github.com/arvados/rnaseq-cwl-training
Also useful for setting up vscode: https://github.com/arvados/arvados-vscode-cwl-training

• Which reminds me a list of resources for development like Benton, etc might be useful
  And his slide deck for additional topics: https://docs.google.com/presentation/d/10-ecKJqEWlnfS5mqe4IU95k6fnvS-kUEePehwXWgQY0/edit?usp=sharing (this is only shared with Bruno since it isn't publicly released)

[kinow]

Hi @swzCuroverse !

I think at the beginning you should explain the general difference between a tool, workflow and expression tool. I think that is what you have in the Basics but just in case I am putting this here

Agreed 👍

Other topics that might be of interest include:

Excellent list of resources! We can include most of these items under Tutorials & How-Tos, and the good practices definitely needs to be put somewhere well visible to users. In the last Biocommons workflows meeting (Australia) there was a poll about workflows after the Janis presentation, where many participants pointed they'd like to know where to find best practices about workflow tools (not only CWL).

And my bad for not including FAIR! It completely slipped my mind, great point.

Thank you!

[skanwal]

Thanks @kinow - this is a great and comprehensive list of contents, resources and to-do's.

My two cents are:

• One of things that I felt was under documented previously is input schemas and SchemaDefRequirement
• data creation/manipulation at runtime (for example at specific mount paths)
• expression Lib (i.e. Advanced CWL expression JavaScript, using external libraries and expressionLib #126 as pointed above)
• usage of arguments VS inputs. I found it confusing at the start to understand when and where to use these.

[kinow]

Hi @skanwal

My two cents are:

+1 to all your points! I also struggled with arguments vs. inputs, and even now sometimes I use an input in some cases where an argument would be simpler 😅

Keep sending the suggestions if you have any more. I received some feedback from @tetron and @swzCuroverse this week, and I'll add it to the top comment later (updating the PR for the theme/sphinx at the moment).

Thank you!!!
Bruno

[kinow]

I have pinned this issue to attract more attention to users coming to report issues.

Also added links to each section of the current User Guide in bold. You can search by "episode" if you prefer too. I called every page an episode, even though not all were episodes (e.g. Extras > YAML is "episode YAML") so that searching for it is easier here 👍 cc @tetron

[kinow]

Added the Wiki about using TypeScript with CWL that @alexiswl presented yesterday in the APAC-EMEA meeting. I added it under Advanced Topics as suggested by @skanwal , but maybe it should be duplicated or mentioned in Recommended Practices too (I would recommend it for users maintaining long or complicated JS expressions).

[skanwal]

+1 for mentioning under both sections @kinow

[mr-c]

Added the Wiki about using TypeScript with CWL that @alexiswl presented yesterday in the APAC-EMEA meeting. I added it under Advanced Topics as suggested by @skanwal , but maybe it should be duplicated or mentioned in Recommended Practices too (I would recommend it for users maintaining long or complicated JS expressions).

This needs review before becoming a "recommended practice". Transpiling is cool, but how might that affect re-use?

[kinow]

Added the Wiki about using TypeScript with CWL that @alexiswl presented yesterday in the APAC-EMEA meeting. I added it under Advanced Topics as suggested by @skanwal , but maybe it should be duplicated or mentioned in Recommended Practices too (I would recommend it for users maintaining long or complicated JS expressions).

This needs review before becoming a "recommended practice". Transpiling is cool, but how might that affect re-use?

Yup, maybe Advanced Topics would be a better home for this then. I mentioned to @alexiswl , that would be nice to have some of pros & cons (the pros being pretty much implicit), e.g.:

• It hinders portability within a team of CWL workflow developers, if any one of them is not familiar with TypeScript
• It also hinders portability between workflow manager tools, as translating CWL JS expressions into other standards/languages like WDL or Nextflow's DSL is complicated; adding an extra level makes it harder
• It can also create issues if there are issues with tsc or jest versions, or if there are extra tools involved (Babel, Vite, esbuild, etc)

[alexiswl]

the pros being pretty much implicit

I’m not sure all of the pros are implicit, they certainly weren’t for me until I started playing around. And this I assume is the case for many devs unfamiliar with TypeScript.

It hinders portability within a team of CWL workflow developers, if any one of them is not familiar with TypeScript

Yes and no to this one. I think TypeScript is much easier to read than ES5 JavaScript, but also very easy to pick up if a user knows JS. So, if a user is not familiar with TypeScript, I doubt that they’re particularly familiar with JavaScript. An alternative argument is that the TypeScript methods might mean that users use the JavaScript expressions of CWL more extensively, which pushes out developers unfamiliar with the TypeScript syntax.

It also hinders portability between workflow manager tools, as translating CWL JS expressions into other standards/languages like WDL or Nextflow's DSL is complicated; adding an extra level makes it harder

Yes and no to this one too. I think TypeScript would be very consistent on its methods of transpiling to ES5 code so that could be a benefit. I’m not familiar with the processes on how this would be converted to a language like WDL. I think this would be an issue with JavaScript expressions in general and should be noted in the user guide if a developer wants to make their workflows language agnostic.

It can also create issues if there are issues with tsc or jest versions, or if there are extra tools involved (Babel, Vite, esbuild, etc)

This is where a package-lock.json file would be useful and should make the transpilation and testing reproducible.

[kinow]

Hi @AlexisW, good pointsl ! I've created the issue common-workflow-language/common-workflow-language#227 to move the discussion over there 👍

[swzCuroverse]

@kinow Can you help me figure out which parts are left "to-do"?

[kinow]

@kinow Can you help me figure out which parts are left "to-do"?

Sure. I think the unchecked boxes have not been addressed here as we decided to prioritize the new structure. I left notes to myself and to future developers in the source code. So the Markdown files must contain comments (start with %), some with TODO markers that contain links to issues/pull requests/discussions. I can try to compile or review a list later (ran out of time to have fun working on CWL this weekend unfortunately 😥 👋 )