Built-in support for cloud technologies and containers?

[hjoliver]

(Roadmap placeholder issue)

The core Cylc sites are getting next-gen HPCs and expect increasing use of cloud tech in the future.

Cylc can of course "run in the cloud", even with dynamic resource scaling via an HPC workload manager such as PBS.

However, do we need to have built-in support for (e.g.):

• running some tasks on cloud platforms, via a local scheduler
• (similarly, workflows that distribute tasks or sub-worklfows across multiple Momentum Partner sites?)
• using cloud-native "job runners" such as kubernetes?
  • our need for a shared run-directory per install target may be a problem here?
• running containers as tasks

First we need to determine requirements from our core community.

---

Short-term easy wins:

• Release Cylc Docker containers and develop small changes to better support cloud deployments
• Document roll-your-own use of cloud platforms with Cylc

---

longer-term (via @oliver-sanders ):

• "Native" cloud - aka advanced deployment patterns, Helm charts and all that (requires re-tooling, cloud knowledge + access, etc).