Proposal: WDL v1.1

[jdidion]

This is a proposal to create a WDL v1.1 that contains some or all of the non-breaking changes from development. I am making this proposal for the following reasons:

• We currently have no timeline for releasing 2.0.
• Execution engines differ on the extent to which they support development.
• We have no compliance test suite nor any procedures or policies around determining whether an engine is compliant, so even if we were to release 2.0, it will be difficult for users to know the extent to which each engine supports the new syntax/features.
• I know of several groups that are using development now, largely to gain access to some of the new standard library functions.
• Every time we update our engine to be compliant with changes to the spec, it potentially breaks workflows using development features, which is frustrating for users.

I have tried to summarize all of the changes that have occurred between v1.0 and development here: https://github.com/openwdl/wdl/blob/d7ecb7392d1cafdd270fdfe7460706140203cf9b/versions/Differences.md

I suggest that we quickly discuss which of these changes to include in a v1.1. Once we are agreed, I volunteer to create a PR with the 1.1 spec. I also volunteer to create an ANTLR4 grammar for v1.1. My hope is that we can ratify 1.1 by the end of 2020.

I will start off the discussion with the following straw-person proposal of what to include:

• Relative imports
• All new standard library functions: min, max, suffix, quote, squote, keys, as_map, as_pairs, collect_by_key, sep
• Support container as an alias of docker in the runtime section
  • Allow image names to be in URI form (e.g. docker://repo/img:tag), and assume docker by default
• JSON input/output syntax
• Syntax for overriding runtime and hint attributes in the input JSON
• The None value
• The new notation for struct literals
• The allowNestedInputs for allowing unsatisfied task inputs
• The new call after syntax
• The new pass-through syntax for call inputs

We should also include all of the cleanup/clarifications that have happened since 1.0, including:

• Unsatisfied task inputs
• Handling of optional file outputs
• The semantics of the officially supported runtime keys
• All the little typo fixes and clean-up of code examples (these can be discovered by a diff between the development and 1.0 specs)

We should deprecate, but not remove, the following (and state explicitly that these features will be removed in 2.0):

• Use of file:// imports
• The object type, object literal syntax, and the read/write_object(s) functions
• The sep, true/false, and default placeholder options
• Use of non-POSIX ERE regular expressions in the sub function
• Use of unsatisfied task inputs without the presence of allowNestedInputs=true
• Use of arbitrary keys in the runtime section
• Use of docker key in the runtime section.
• That docker/container is optional (currently Clarified Runtime and Added Hints #315 states that container will become required in 2.0)

The following will not be added in 1.1:

• The Directory type: still half-baked, and probably needs some additional functions in stdlib to be useful
• The hints section: with arbitrary keys still allowed in runtime, the addition of hints would be confusing and offer no benefit; there's also still a debate about what types of values are allowed in the hints section

[jdidion]

I propose that we schedule a vote on this proposal for Monday, November 23rd. Assuming the vote passes, I will commit to having the PRs in place by Monday, December 7th. I would like the rest of the governance board to commit to reviewing and merging the PRs no later than Friday, December 18th.

[jdidion]

Please note that I specifically did not include addition of the Directory type, even though it is a non-breaking change. This is my own bias, because it is actually a major and complicated change that we need to deal with in cloud-land. But if everyone else wants to include this change, I won't oppose it.

[jdidion]

Thanks @mlin - I did miss #162. The other breaking change is that in development runtime does not allow arbitrary keys, which would still be allowed in 1.1. But generally you're correct.

[jdidion]

We can also get #365 in there if we can get it passed before the 23rd.

[rhpvorderman]

As posted below. I think releasing a 1.1 creates more troubles than it is worth. But we could also leave Directory types out of 2.0 and add those features in 2.1 or another 2.x version.

[illusional]

Directories are so useful for RNA Seq analysis. Can’t we leave them in and engines that don’t implement them are just not implementing the whole spec.

[illusional]

I don’t mind interpolation options staying, but selfishly I want the sep function.

[mlin]

I opened a couple of late discussion points on #315 (runtime/hints); I'm moderately concerned about going off half-cocked there, but nothing we shouldn't be able to work through (if we can all volunteer the time 😅)

[jdidion]

Looks like there's a reasonable proposal - allow URI-formatted image names and assume Docker by default. If we can get consensus on that (at least for v1.1 - we can still change it later in development), then I think we can move forward with a vote on this proposal.

[cjllanwarne]

I'd prefer we only make non-breaking changes, ie leave the "remove object" stuff out until 2.0 (and indeed all of the other "removals" section)

[cjllanwarne]

If we do want to make breaking changes and they'll be super-useful to users NOW, I suggest we call this "intermediate" version 2.0 anyway, and leave the remaining changes we had previously scheduled for 2.0 until a 3.0.

[geoffjentry]

If there are any breaking changes, by definition this would be 2.0, not 1.1

[rhpvorderman]

We can also have a 2.1 later right? Is there a need to break things in 2.0 already? 😲

The move from 1.0 to 2.0 has taken a long time. I feel that we should make sure that a 3.0 is not needed in the foreseeable future. So if we want to break things from 1.0 we should do it as soon as possible. The quicker this is done the better it is. If these important changes (removing objecg etc.) are not followed trough on now, but instead wdl 1.0 is improved onto wdl 1.1 1.2 etc. What will happen is:

1. We will have to live longer with annoying features of WDL that we want to break.
2. More and more WDL 1.x workflows will be built making the transition more work.
3. Due to 1 and 2. A parallel wdl 1.x and 2.x universe will start to emerge. Increasing maintenance for the developers of execution engines with more than 2x. (There are significant overhead costs as well too maintaining two parallel branches).

I do not want another python 2.x vs python 3.x situation.

I propose we go to 2.0 now. Break all the stuff that needs to be broken, and do not backport any usability features to 1.x. This way we do not have to update engines, since they can already run 1.0. Also there is incentive for users to switch to 2.0. EDIT: I should have read the proposal more thoroughly. I think a 1.1 is a good in between step. Especially since there will be deprecation warnings. I only hope that we will never release a 1.2.

In the foreseeable future we probably do not need to break stuff any more. So we can iterate faster on backwards-compatible updates 2.1, 2.2 etc.

[jdidion]

Is there a need to break things in 2.0 already?

If nothing is broken, then it would not necessitate a major version increment. If there are breaking changes, it will be 2.0. If there are not, it will be 1.1.

... if we want to break things from 1.0 we should do it as soon as possible.

There is nothing to say that progress on new non-breaking features can't happen concurrently in 1.x versions with breaking changes in development/2.0. This is mostly a problem for the developers of the runtime engines, all of whom (I believe) are represented on the governance board.

For community/consensus-driven specifications, there is always a tension between iterating too slowly - thus leaving lots of cool features that everyone wants "in development" and implementors making different decisions about which features they'll support immediately and which they'll hold off on until the official version is released - and iterating too quickly - thus producing many different versions that have to be supported by implementors.

My personal feeling is that we should decide on a release cycle and stick to it; and that each release should have a pre-defined support lifetime after which it is acceptable for runtime engines to drop support. Whatever new features are ready near the end of each cycle go into the next version. I am in favor of either semi-annual or annual releases, with a 2-year support period for each version.

In the foreseeable future we probably do not need to break stuff any more.

This is ideal, but I also don't think the evolution of a language should necessarily be constraint by a fear of breaking changes.

The one big bugbear I still see in WDL is the type system. If that's a change we want to take on for 2.0, great. If not, then I think we'll eventually need a 3.0. Other than that I can't see anything else in the language we'll need to break to be able to move it forward.

[rhpvorderman]

My personal feeling is that we should decide on a release cycle and stick to it; and that each release should have a pre-defined support lifetime after which it is acceptable for runtime engines to drop support. Whatever new features are ready near the end of each cycle go into the next version. I am in favor of either semi-annual or annual releases, with a 2-year support period for each version.

I am also in favor of a release cycle. Should we make this into a separate discussion? The last one fizzled out.

The one big bugbear I still see in WDL is the type system. If that's a change we want to take on for 2.0, great. If not, then I think we'll eventually need a 3.0. Other than that I can't see anything else in the language we'll need to break to be able to move it forward.

A 3.0 in the short-term is not desirable in my opinion. So I think we can take it on for 2.0. We can have a 3.0 eventually if issues arise, as it is impossible to plan for the unknown. But if there are any pressing issues now we should prefer to fix them now rather than later.

[jdidion]

This proposal is open for voting. The original proposal has been updated to reflect the results of discussion. Please quickly re-read the proposal and then leave a comment with your vote.

[illusional]

If the sep(String, Array[String]) function made it into v1.1: 👍
It's not explicitly included, but it's not dependent on interpolator options being removed.

[geoffjentry]

I'm 👍

@illusional to be clear, does the if ... mean that you are 👎 otherwise?

[cjllanwarne]

👍 from me too.

[mlin]

👍

[rhpvorderman]

👍. Great work @jdidion !

[jdidion]

I'm going to consider this passing and move forward with the PR.

[jdidion]

PR submitted #428