Can we use docstrings in pypyr

[blaisep]

I honestly don't know if this may be more of a yaml question... I wonder if we can insert docstrings in the yaml files?
My motivation is that docstrings can/should include rST which can then be processed by powerful set of python tools

So, it would then be possible to automate the documentation for pipelines....

[yaythomas]

Yaml only allows single line comments with #.

Nothing stops you from declaring your own key though - just make sure it doesn't clash with a step-group name. Notice the block scalar style | indicator for multi-line strings keeping \n:

_docstring: |
  Several lines of text,
  with some "quotes" of various 'types',
  and also a blank line:

steps:
  - name: pypyr.steps.echo
     in:
       echoMe: hello

It is then up to you to parse that out of the pipeline when you generate your help docs.

import pypyr.loaders.file as pypyr_file_loader
pipe_def = pypyr_file_loader.get_pipeline_definition("pipeline_name", Path.cwd())
print(pipe_def.pipeline['_docstring'])

[yaythomas]

Not so incidentally, something that's been brewing on the back-burner for ages now is introducing a metadata sort of section, like:

_meta:
  author: firstname lastname
  help: |
    long help text here
    follow what docstring format you like
    some_other_field: whatever

steps:
  - name: pypyr.steps.echo
    in:
       echoMe: hello

And then expose that maybe via

$ pypyr my-pipeline --help
long help text here
follow what docstring format you like

# or maybe even
$ pypyr my-pipeline --meta
author: first name lastname
some_other_field: whatever

I remain ambivalent about whether this would be truly useful or not, which is why I've not prioritised it. But I could well be convinced...

This would effectively introduce a "reserved word" for the _meta section, since this would clash/break any user pipelines that happen to have a custom step group called _meta. Not that this is super-likely, mind, but just as a general principle I'm cautious about introducing "reserved" keywords into the pipeline or context so it doesn't turn into a free for all.

[blaisep]

As someone raised in Software Defined Networking, I really like the idea of being able to have a structured link between control plane and the data plane.

[blaisep]

btw, this has a long history in file systems... VMS, NTFS and Apple's many file systems have all included resource and data forks for every file.

[yaythomas]

Assuming there were a meta section, what sort of data/keys/properties would you expect to see there, or find useful?