Add new commands to pipeline args

[annhov]

I'd like to add new commands in the pipeline initialization part, i.e extend this part of the pipeline file:

I've already modified the kfp/operator.py script and added the required lines to the arguments_list. After building elyra using setup.py I still can't see the newly added parts during the pipeline execution. Looks like operator.py is somehow being replaced while running the notebook in the custom runtime image as the commands are added to the pipeline even if elyra is not installed in the image.

Is there anything else that should be modified in the source code to be able to change the ExecureFileOp behavior?

[akchinSTC]

Hi @annhov, do you by chance have a fork/branch of the changes we can look at?

[annhov]

Hi @akchinSTC, actually no. What I did was clone the whole project and modify only this part of ExecuteFileOp's init from elyra/elyra/kfp/operator.py:

        if "arguments" not in kwargs:
            """If no arguments are passed, we use our own.
            If ['arguments'] are set, we assume container's ENTRYPOINT is set and dependencies are installed
            NOTE: Images being pulled must have python3 available on PATH and cURL utility
            """

            common_curl_options = '--fail -H "Cache-Control: no-cache"'

            argument_list.append(
                f"mkdir -p {self.container_work_dir} && cd {self.container_work_dir} && "
                f"echo 'Downloading custom files' && "
                f"curl {common_curl_options} -L https://www.python.org/static/apple-touch-icon-144x144-precomposed.png" # the url is modified
            )

Then, created a new runtime image using this Dockerfile:

FROM jupyterhub/k8s-singleuser-sample:1.2.0

ARG ELYRA_EXTRAS=[kfp-tekton]

USER root
RUN apt-get update && apt-get install -y curl ffmpeg libsm6 libxext6
RUN conda install python=3.8

COPY ./elyra ./elyra/
WORKDIR elyra
RUN python3 -m pip install -r build_requirements.txt && python3 setup.py install && python3 -m pip install -r elyra/pipeline/kfp/requirements-elyra.txt

WORKDIR /tmp
COPY docker/start-elyra.sh /usr/local/bin/start-elyra.sh
RUN chown -R root:root /usr/local/bin/start-elyra.sh
RUN chmod 0755 /usr/local/bin/start-elyra.sh
CMD ["/usr/local/bin/start-elyra.sh"]

I can confirm that the right version of operator script is installed in conda environment, however the notebook run with this runtime image starts with the following lines:

[akchinSTC]

Since it looks like you're running a development version of elyra, you can probably simplify the RUN python3 -m pip install -r build_requirements.txt && python3 setup.py install && python3 -m pip install -r elyra/pipeline/kfp/requirements-elyra.txt in your dockerfile to just use our Make target make install. You can probably remove ARG ELYRA_EXTRAS=[kfp-tekton] as well since this doesnt look like it does anything and our make target will install tekton by default.

[annhov]

@akchinSTC, doing make instead of installing via setup.py helped. Thank you.

[ptitzler]

fwiw I've changed the discussion thread category to Q&A and marked Alan's response as answer. This will hopefully make it easier for others to find the solution, should they try to accomplish something similar.