Global variables

[mtazzari]

First of all: congrats for the amazing tool you have developed!
I was looking at Pythonic and elegant ways to develop pipelines...and I've found pypyr!

A question I have is whether it is possible to define global variables. Use case is as follows:

globals:
  varA: hello
  varB: 2
  default_kwargs:
    task1:
      x: 1
      y: 2

steps:
  - name: mystep
      in:
        - taskname: task3
        - A: $varA
        - B: $varB
  - name: mystep
      in:
        - taskname: task1
        - kwargs: $default_kwargs.task1
        - x: 55
        

In my case, I'm writing my own mystep so that it takes the kwargs and runs a task from a specific library with the input kwargs. In real case, the tasks have many parameters so that it'd make sense to define global defaults and override just a few parameter values in different steps.

Do you think it is possible to do that?

Many thanks in advance!

[yaythomas]

thanks for you kind words @mtazzari, much appreciated 🙌 !

it's absolutely possible to do what you want. in pypyr the easiest is to use formatting substitution expressions - these support complex types like your own dict/kwarg style arguments.

I normally use the pypyr.steps.default step to set "global" values in context that I want to use later in the pipeline. You can also use pypyr.steps.contextsetf - the only difference is that default only sets the values if they don't exist already, which is handy if you want to combine this with something like the pypyr.parser.keyvaluepairs parser to allow the user to override some of the defaults from the cmd line. contextsetf by comparison, always set the values, overwriting them if they exist already.

So, just taking your quick example:

# ./mypipeline.yaml
context_parser: pypyr.parser.keyvaluepairs
steps:
  - name: pypyr.steps.default
    in:
      defaults:
        varA: value A
        varB: value B
        varC: value C

        default_kwargs:
          task1:
            x: 1
            y: 2
            z: '{varC}'

  - name: mystep
    in:
      mystep:
        taskname: task3
        A: '{varA}'
        B: '{varB}'


  - name: mystep
    in:
      mystep:
        taskname: task1
        kwargs: '{default_kwargs[task1]}'
        A: '{varA}'
        B: '{varB}'

So this you can run like this:

$ pypyr mypipeline

In this case it will just use the default values.

If you want to override, you can do:

$ pypyr mypipeline varA="value from cmd line" varB=arb

in which case varA and varB will be the value you set from the cmd line, rather than the defaults set in the pipeline.

Notice that you can also nest formatting expressions - so I arbitrarily added an extra z input to your task1 which gets its value from varC.

Hope this helps?

[edit]

Of course, depending on exactly what you're doing, but since you're writing your own step you don't even have to pass kwargs. in the step you could just do this:

def run_step(context):
    task_name = context['mystep']['taskname']
    default_kwargs_for_this_step = context['default_kwargs'][task_name]

[mtazzari]

hey @yaythomas that's amazing, thanks very much for the super quick and extensive reply!
It's exactly what I want to do - indeed while waiting for your answer I was studying pypyr docs and found the default step.
But thanks for clarifying the difference with contextsetf.

Re the edit: that is very similar to what I want to do. The only difference is that I'd like to "update" the default kwargs with the kwargs I specify inside the task. Taking your example I agree I don't even need to pass the default kwargs, which is super neat:

# ./mypipeline.yaml
context_parser: pypyr.parser.keyvaluepairs
steps:
  - name: pypyr.steps.default
    in:
      defaults:
        varA: value A
        varB: value B
        varC: value C

        default_kwargs:
          task1:
            x: 1
            y: 2
            z: '{varC}'

  - name: mystep
    in:
      mystep:
        taskname: task3
        A: '{varA}'
        B: '{varB}'


  - name: mystep
    in:
      mystep:
        taskname: task1
        kwargs:
          x: 30
        A: '{varA}'
        B: '{varB}'

and the run_setup would be something like:

def run_step(context):
    task_name = context['mystep']['taskname']
    default_kwargs_for_this_step = context['default_kwargs'][task_name].update(context.get_formatted('kwargs'))

[yaythomas]

I appreciate you probably just dashed off a rough and ready example there, so I don't mean to preach, but if you're sure you're wanting to update the original values for default_kwargs then yes, dict update() is the way to go. In this case subsequent steps will use the new "global" values that you just updated. (in your example, all subsequent steps will take x == 30 as the default, rather than x == 1.

If what you're wanting to do is instead use default_kwargs, combine with any overrides from the step's kwargs, without overwriting the originals, then you'll either have to copy the dict, or do something like use a ChainMap.

from collections import ChainMap


def run_step(context):
    # raise user friendly err if mystep config not there
    context.assert_key_has_value(key='mystep', caller=__name__)

    mystep_formatted_input = context.get_formatted('mystep')
    task_name = mystep_formatted_input['taskname']

    global_default_kwargs = context.get('default_kwargs')
    default_kwargs_for_this_step = None

    if global_default_kwargs:
        default_kwargs_for_this_step = global_default_kwargs.get(task_name)

    combined_effective_kwargs = ChainMap(mystep_formatted_input.get('kwargs'),
                                         default_kwargs_for_this_step)

    print(task_name)
    if combined_effective_kwargs:
        for arg in combined_effective_kwargs.items():
            print(arg)
    print('--------------')

I added some safety checks for None so the step will work with or without mystep.kwargs or the global default_kwargs existing. And just a usage tip, get_formatted() works with keys directly off the root of context, not nested keys, so I generally find it easiest to do the get_formatted right at the top and then just work with the formatted result for the rest of the step's code.

[mtazzari]

Totally agree, thanks for expanding on it!
Indeed, I just posted a quick one-liner to show the idea but it definitely makes much more sense to avoid modifying the original defaults values. That makes debugging much easier and keeps the pipeline more stateless.
Thanks for the implementation!

[mtazzari]

Just writing here for your reference:
Actually, I was wondering if the step I am developing could be a step of general interest (unless it is already possible with pypyr). On very generic terms, it allows to call any function in any python package installed in the environment, with the possibility to define global default kwargs and to override them on a step-basis, with the behaviour we just discussed above.
Example:

- name: mystep
    in:
      mystep:
        package: numpy
        function: sum
        args: [1, 2]
        kwargs:
          axis: 30
          y: 10

[yaythomas]

hey, that's a good idea!

If you're specifically interested in a return value, you can actually more or less do this already using py strings, and to import numpy use pyimport. . .

  - name: pypyr.steps.pyimport
    in:
      pyImport: import numpy

  - name: pypyr.steps.contextsetf
    in:
      args: [[0, 1], [2, 3]]
      kwargs:
        axis: 0
        keepdims: False
      contextSetf:
        sum_of_stuff: !py numpy.sum(args, **kwargs)

  - name: pypyr.steps.echo
    in:
      echoMe: sum is {sum_of_stuff}

But this is not as intuitively user friendly as what you're suggesting, for sure, so what you're talking about could well be of general interest!