ecoscope-workflows

This repository contains 5 distinct Python packages. Of those, the two contained under src/ represent the core logic developed in this repo:

src
├── ecoscope-workflows-core
└── ecoscope-workflows-ext-ecoscope

• ecoscope-workflows-core is a code generator that provides a specification for defining workflows (i.e. DAGs) of Python callables (known as @tasks) in YAML, as well as a CLI for compiling these YAML DAG definitions into installable packages. The package generated by this compilation process provides both REST API and CLI entrypoints for running the specified workflow.
• ecoscope-workflows-ext-ecoscope is a task extension package which, when installed, registers additional @tasks with the core compiler that wrap logic provided by ecoscope.

The remaining 5 packages in this repo are examples of the types of workflow packages which can be generated by this framework. The structure of each subdirectory in the examples directory adheres to the following format:

examples
└── <EXAMPLE-NAME>
    ├── ecoscope-workflows-<EXAMPLE-NAME>-workflow
    ├── spec.yaml
    └── test-cases.yaml

where:

• EXAMPLE-NAME is the workflow author's selected name for this example
• spec.yaml is the YAML specification defining the workflow
• test-cases.yaml defines one or more test cases, inclusive of example parameters and expected outputs, which can be used to test the workflow after it has been compiled.
• ecoscope-workflows-<EXAMPLE-NAME>-workflow is the dynamically generated package. Note: All code in this subdirectory is machine-generated.

Development

Development functions for all of packages in this repo can be run via pixi from the repo root. To do so:

1. Install pixi
2. From the repo root, call:

   $ pixi run --manifest-path pixi.toml -e default <TASK>

   where <TASK> is one of the tasks defined in the repo root pixi.toml.

Tasks defined in pixi.toml include logic for:

• (re)compiling example workflows
• linting, formatting, and type checking
• pytest
• building and running docker containers
• building and releasing conda packages from the packages defined in this repo (and their dependencies)

Note: For the most part, these pixi tasks wrap logic defined in the dev/*.sh scripts in this repo. If you prefer, you may call them directly.

What environment(s) are these tasks run in?

Every package in this repo has it's own pixi manifest file (either pixi.toml or pyproject.toml), that defines dependencies, tasks, and environments for the project. These docs on environment vs. project are a useful reference for those not used this project-based approach to environments.

Note: The fact that we are managing multiple projects in the same repo means that using the --manifest-path option to explicitly declare which project you are using for a particular invocation tends to be a good idea.

When am I running an editable install of the src/ packages?

There are two flavors for how the ecoscope-workflows-core and ecoscope-workflows-ext-ecoscope (collectively, the src/ packages) are installed in the various pixi project environments:

1. In the context of testing the src/ packages themselves, the environments associated with these projects use editable installs for the src/ packages.
2. For the examples project environments, the src/ packages are not installed in editable mode but rather from an ephemeral local "release" (i.e. conda package) built from the src/ packages. This choice was made to align dependencies management in the examples with the way in which released workflows manage dependencies. The trade-off for this "realism" is an additional step in the development process; namely, re-building the local "release" if new changes from src/ are required for testing the example workflows. This is done automatically in the CI testing of the examples, so you can be confident that the CI is always being run with the latest changes.

Note: Pixi tasks support caching, and we currently do use caching to attempt to avoid unecessary re-builds of the local src/ releases (in the context of local development, not CI). However, please be aware that tuning this logic remains a work-in-progress and further work needs to be done to ensure that these cached builds are being re-built at the appropriate times.

Release

To release conda packages for ecoscope-workflows-core and ecoscope-workflows-ext-ecoscope:

1. Open and merge a PR that bumps the context version for both recipes in publish/recipes/release/*.yaml to the desired release tag, if the ecoscope (core) version has changed in src/ecoscope-workflows-ext-ecoscope/pyproject.toml, then this will also need to be bumped in the recipt for ecoscope-workflows-ext-ecoscope

   Note: To get examples CI tests to pass, you will also need to build a release locally with the build-release pixi task, and then recompile all examples to get the new release tag reflected in the lockfiles for all examples. We may be able to remove this requirement eventually, as we improve our release flow, but for now this is required.

2. Rebase your local clone to be equal with main (following the merge of the context version bump PR), and then push a new tag to the remote, e.g.:

   $ git tag v0.0.2
   $ git push origin --tags

3. The publish.yml github workflow will then build and push a new release to both GitHub Releases and https://prefix.dev/channels/ecoscope-workflows