Skip to content

Commit 3cc5620

Browse files
committed
Add support for dynamic pipelines to the Vertex orchestrator
1 parent d8bd68a commit 3cc5620

File tree

5 files changed

+518
-249
lines changed

5 files changed

+518
-249
lines changed

docs/book/how-to/steps-pipelines/dynamic_pipelines.md

Lines changed: 3 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ description: Write dynamic pipelines
55
# Dynamic Pipelines (Experimental)
66

77
{% hint style="warning" %}
8-
**Experimental Feature**: Dynamic pipelines are currently an experimental feature. There are known issues and limitations, and the interface is subject to change. This feature is only supported by the `local` and `kubernetes` orchestrators. If you encounter any issues or have feedback, please let us know at [https://github.com/zenml-io/zenml/issues](https://github.com/zenml-io/zenml/issues).
8+
**Experimental Feature**: Dynamic pipelines are currently an experimental feature. There are known issues and limitations, and the interface is subject to change. This feature is only supported by the `local`, `kubernetes`, `sagemaker` and `vertex` orchestrators. If you encounter any issues or have feedback, please let us know at [https://github.com/zenml-io/zenml/issues](https://github.com/zenml-io/zenml/issues).
99
{% endhint %}
1010

1111
{% hint style="info" %}
@@ -265,26 +265,11 @@ When running multiple steps concurrently using `step.submit()`, a failure in one
265265
Dynamic pipelines are currently only supported by:
266266
- `local` orchestrator
267267
- `kubernetes` orchestrator
268+
- `sagemaker` orchestrator
269+
- `vertex` orchestrator
268270

269271
Other orchestrators will raise an error if you try to run a dynamic pipeline with them.
270272

271-
### Remote Execution Requirement
272-
273-
When running dynamic pipelines remotely (e.g., with the `kubernetes` orchestrator), you **must** include `depends_on` for at least one step in your pipeline definition. This is currently required due to a bug in remote execution.
274-
275-
{% hint style="warning" %}
276-
**Required for Remote Execution**: Without `depends_on`, remote execution will fail. This requirement does not apply when running locally with the `local` orchestrator.
277-
{% endhint %}
278-
279-
For example:
280-
281-
```python
282-
@pipeline(dynamic=True, depends_on=[some_step])
283-
def dynamic_pipeline():
284-
some_step()
285-
# ... rest of your pipeline
286-
```
287-
288273
### Artifact Loading
289274

290275
When you call `.load()` on an artifact in a dynamic pipeline, it synchronously loads the data. For large artifacts or when you want to maintain parallelism, consider passing the step outputs (future or artifact) directly to downstream steps instead of loading them.

src/zenml/integrations/aws/orchestrators/sagemaker_orchestrator.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1050,10 +1050,10 @@ def _wait_for_completion() -> None:
10501050
metadata=metadata,
10511051
)
10521052

1053-
def launch_dynamic_step(
1053+
def run_isolated_step(
10541054
self, step_run_info: "StepRunInfo", environment: Dict[str, str]
10551055
) -> None:
1056-
"""Launch a dynamic step.
1056+
"""Runs an isolated step on Sagemaker.
10571057
10581058
Args:
10591059
step_run_info: The step run information.

0 commit comments

Comments
 (0)