Centralise Cylc CLI Functionality

[oliver-sanders]

At present the cylc command lives in Cylc Flow.

As of cylc/cylc-flow#3413 it is possible for other Cylc components to add their own Cylc commands.

It would probably make more sense for the cylc command to live in a common module which the others could used in turn. This common module should also contain the cylc version command which should now provide the version of the Cylc "super package", and, on-request a list of installed components and their versions. Note the list of installed components could be obtained from "entry points" in some way, this would work for both pipy and conda installations.

This "common module" could be:

• (yet) another repository / pypi project.
• A git submodule.
• Something else?

[hjoliver]

With entry points, another repository would be reasonable.

Git submodules (for "shared components"?) are interesting but I haven't worked with them enough (or at all) to have a good feel for whether or not we should use them for this or anything else. (Maybe others have?).

[kinow]

Interesting, I think I haven't seen this done in another Python project. Maybe @sgaist will have an opinion on this too, or maybe he's seen entry points used this way?

Note that when we started to move to setuptools/setup.py, namespaces, cylc version - if I recall correctly - was "too smart". It had an environment variable I think, and also tried to give extra information. This didn't work with Conda, virtual envs, etc.

So given the number of modules, and the combinations that people can have for their environments (pip, conda, maybe some will want RPM or DEB packages, or even pyinstaller/spack/pipx/poetry/etc), I would only check if this change wouldn't become an issue later.

Another thing that needs to be checked is circular dependency. I think it won't happen as your suggestion would move only cylc to that module. But it would be complicated if this new project had a dependency on cylc-flow, and if cylc-flow also had to have a dependency on this project.

Other than that, +1

[oliver-sanders]

I think the new cylc version should be pretty straight forward:

#!psudocode

print('cylc', __version__)
for package in entry_points:
    version = package.__version__
    if is_git_repo(package.__file__):
        version += '-' + '$(git describe --abbrev=4 --always)'
    print(package.__name__, version)

[oliver-sanders]

I think I haven't seen this done in another Python project

No me neither. I think most of the time you have one "core" package and lots of "plugins". With Cylc it's a bit different as we are installing different bits of a system on different machines so we can't rely on the "core" Cylc Flow package always being installed.

Note: we may well chop Cylc Flow down further to avoid having to install the entire package on remote hosts when all job hosts really need is cylc message etc.

[kinow]

Note: we may well chop Cylc Flow down further to avoid having to install the entire package on remote hosts when all job hosts really need is cylc message etc.

Ah, that sounds even more interesting. Maybe if you have time later to try a proof of concept of how it would work, then we can have a look and discuss more. I'd be happy to have some fun trying it out, and testing locally too 👍

[kinow]

(Another thing to note, add a post-requirement task to update our Conda recipes, and add this one to the Conda Forge too)

[oliver-sanders]

We have some static resources which could do with being common as well e.g. the Cylc logo.

[sgaist]

@kinow Do you mean you are looking for a project that extends a "core" Click command group with further commands ?

[kinow]

@kinow Do you mean you are looking for a project that extends a "core" Click command group with further commands ?

Hi @sgaist , more a case where the core command is in a separate repository. The cases I know have the core command as part of the main utility/repository (e.g. pytest, `.

But it could be that I am simply being too dense for being more used/familiar to conda list or pip list and preferring to avoid extra layers. Also as I am experimenting more using Cylc with Docker, Singularity, and thinking in how it could be used in Cloud environments, Kubernetes, etc, I see less need to have cylc version with list of dependencies.

But if having cylc version is something that will be helpful to users/site admins, then I won't object, and think the central Cylc CLI would be necessary to avoid the dependency on cylc-flow...

[oliver-sanders]

OK some quick context:

• Cylc users are mostly scientists not developers.
• Users don't necessarily know what PIP, Conda or Docker are, our more technical users often move in different circles.
• Cylc is often used at institutions who install it for their users so users don't actually know how Cylc was installed for them.
• Institutions often don't provide users with tools like PIP, Conda or Docker (prefering to pre-build any environments users need).

Basically pip list just isn't an option for most users. Gotta remember we are not building a tool for developers, some of our users have very good technical skills and we try to provide for them, but many aren't familiar with the command line or even Linux.

Besides I think we need to centralise the CLI anyway:

• To allow partial installation on remote hosts (don't need the whole of Cylc Flow installed on the supercomputer)
• cylc-uiserver should be cylc uiserver
• jupyterhub should be cylc hub

Listing registered packages via entry points should be easy.

[kinow]

I am aware that most our users now are scientists. Many of the changes we have made in the past months make things more complicated for scientists (multiple repositories, no more batteries included, need to use conda or pip to install, etc).

But I think we did these changes for a good reason. And the old cylc version was problematic to migrate due to it being too elaborate (we also had some environment variable that gave some trouble during refactoring).

So there is a counter argument for each of your arguments. I also don't think having a command in front of jupyterhub, or that cylc uiserver is necessary. But we make no progress discussing like this.

If you say that will be best for users, I will have no objection and won't try to block any of these changes 👍

[oliver-sanders]

Many of the changes we have made in the past months make things more complicated for scientists

Scientists won't see these changes, they will just use the installation they are given.

But even crossing off those arguments we are still left with:

• The issue of running Cylc commands in the absence of a full Cylc installation. (e.g. cylc message is pretty much all we need installed on a supercomputer*)
• The issue of users not having access or permissions for commands like pip.

[hjoliver]

@oliver-sanders - another point @kinow made offline (he can correct me if I misrepresent him) is, thinking ahead to looser cloud-based deployments where J-hub might be separately deployed to one host, UI Server(s) to another(s), cylc-flow to others, or whatever, ... where the main CLI command resides is clear enough if it stays with cylc-flow, but otherwise (if it needs to know about all components) not so clear. E.g. if we don't end up modifying J-hub at all (and hopefully we won't) it may not even get installed via Cylc. So how is our cylc --version command going to know about it? so maybe (attrib. @kinow ) this isn't a problem we can really solve centrally.

The only sticking point for me is, it does seem unnecessary to have to install the full cylc-flow package just to get the main cylc command for cylc message on job hosts...

[oliver-sanders]

otherwise (if it needs to know about all components) not so clear

Not really.

If we have installations in different containers that's no problem, we wouldn't expect the installation in one container to know about the installation in another.

Whenever you run a cylc command you are running a Cylc executable installed in some location. cylc --version could just dump out all cylc plugins installed in that location.

if we don't end up modifying J-hub at all (and hopefully we won't) ... So how is our cylc --version command going to know about it?

Easy, like this:

cylc-uiserver/setup.cfg:

[options.entry_points]
console_scripts=
    cylc-hub = jupyterhub.app:main
    cylc-uiserver = cylc.uiserver.main:main
cylc.module=
    uiserver = 1.0.0

[oliver-sanders]

This ticket seems to have generated a lot of discussion which confuses me.

So to quickly explain what I've proposed on the cylc --version front is very simple and would use the entry points mechanism which we already rely on. It should be as simple as this:

cylc-flow/setup.cfg

[options.entry_points]
console scripts=
     cylc-run = cylc.flow.scripts.run.main:main
     ...
module=
     flow = 8.0.0

cylc-uiserver/setup.cfg

[options.entry_points]
console_scripts=
    cylc-hub = jupyterhub.app:main
    cylc-uiserver = cylc.uiserver.main:main
cylc.module=
    uiserver = 1.0.0

cylc/common/__init__.py

def version():
    print(f'Cylc {__version__}'
    for module, version in get_entry_point('cylc.module).items():
        print(f'    {module}: {version}')

We don't actually need the cylc.module entry point (could use package name or other entry point[s]), I've used cylc.module for illustrative purposes.

This is fully independent of PIP, Conda, Docker, whatever, it simply lists the names and versions of and Cylc packages installed in the local Python environment.

Managing stacks, environments and containers is your own lookout, we can't help you there!