Run specific pipelines on VertexAI

[rafpinter]

Hi everyone!

First, thank you for this awesome plugin :)

I have a question regarding the execution of specific pipelines on VertexAI. Is there a way to accomplish this? For example, I would like to run my training pipeline at one time, and the predict pipeline at another.

I tried building a Docker image that receives as input the name of the pipeline I want to execute. The part of the Dockerfile responsible for this is:

ARG _PIPELINE
ENV PIPELINE=$_PIPELINE

CMD if [ -z "$PIPELINE" ] ; then kedro run ; else kedro run --pipeline $PIPELINE ; fi

When I use this Dockerfile to build an image and run the pipeline, it always executes the whole pipeline. I'm not sure if there's an error in my code or if it's not supported by the plugin.

Any help is very much appreciated :)

[marrrcin]

How are you invoking your pipeline? There is no need to change the Dockerfile to run different pipeline,
just use kedro vertexai run-once --pipeline=<name of your pipeline>

[rafpinter]

Indeed, I was simply using the kedro vertexai run-once command. Adding the --pipeline=train parameter worked perfectly. Thank you so much!