RFC: interpreter runtime attribute

[jdidion]

Cross-posted from #404

Currently, it is specified that the task command block is executed by a bash interpreter. Non-bash languages are supported either by heredocs or by putting the commands in an external script and calling that script from the command block.

This proposal is to add the optional interpreter attribute to the runtime section. This keyword will be used to specify the interpreter program (name, version, etc) used to execute the command block. This information can be used by the WDL runtime in several ways:

• Execute non-bash scripts without the need for heredocs or external scripts
• Pre-check to verify that the specified interpreter is available in the runtime environment (container or host system) before trying to execute the commands
• Choose a default runtime environment (container, virtualenv, etc) when a container is not specified

Example:

task python_cat_file {
  input {
    File infile
  }

  command <<<
  import sys
  path = "~{infile}"
  print(f"reading from file {path}", file=sys.stderr)
  with open(path, "r") as i:
    print(i.read())
  >>>

  output {
    String content = read_string(stdout())
  }

  runtime {
    interpreter: {
      name: "python",
      version: "^3.6.0",
      variant: "pypy"
    }
  }
}

The interpreter keyword can take one of the following values:

• String with the interpreter name (e.g. "python") and optional version constraint.

  runtime {
    interpreter: "python ^3.6.0"
  }

  • The interpreter name is case-insensitive. It is suggested to use all lowercase.
  • For a programming language with multiple interpreters available, the language name (e.g. "python") is an alias for the reference/default interpreter variant (e.g. "cpython").
  • If the version is not specified, the default version is used. The default version may be defined in the WDL specification, or it may be specified by the runtime.
• Array[String] with multiple interpreter variants, in order of preference.

  runtime {
    interpreter: ["cpython ^3.6.0", "PyPy3.6 ^7.3.0"]
  }

• Object with the following keys:
  • name: The interpreter name. Required.
  • version: An optional version constraint.
  • variant: An optional interpreter variant.
  • args: An optional Array[String] with arguments to pass to the interpreter when executing the command. The runtime may specify interpreter arguments that are always used and/or not allowed - any arguments specified by the args keyword are merged with the required arguments, and an error is raised if any of the user-specified arguments are disallowed.

  runtime {
    interpreter: {
      name: "bash",
      version: "^5.0",
      args: ["--posix"]
    }
  }

• Array[Object] with multiple Object-style interpreter variants.

Additional rules:

• If no interpreter keyword is specified, the default value is "bash".
• If the interpreter keyword is specified, the host system must provided an interpreter that is compatible with one of the user's specifications, or raise an error if no compatible interpreters are available.
• If the user specifies the container runtime attribute, then one of the specified interpreters must be available in that container.
  • If container specifies an array of containers, then the runtime may (but is not required to) use the availability of one of the specified interpreters within the container as a criteria in selecting which of the containers to use.
• If the user does not specify the container runtime attribute then the runtime may try either or both of the following, in any order. In either case, no other dependencies are guaranteed to exist aside from the interpreter.
  • Execute the commands on the host system by searching for a compatible interpreter
  • Execute the commands in a container of the runtime's choice that provides one of the specified interpreters

[mlin]

When processing the version constraint, how is the engine supposed to discover the interpreter versions available in the container image or the local/execution host?

I'm a little nervous about the portability fragmentation that might result from different engines supporting different subsets of possible interpreters. It seems to me that if a task has highly specific constraints for the runtime interpreter version, it's reasonable to expect it to supply a container with something appropriate (#406 would simplify this in a lot of cases).

Rather than name and version, then, could runtime.interpreter be the path to the interpreter executable (+ arguments)? retroactively defaulting to /bin/bash