Running Argo workflows from a remote machine on Nebari

[rsignell]

I tried out using Argo from the Argo UI, and it worked! Awesome!

Now I'm wondering whether it is possible to launch an Argo workflow on the Nebari kube cluster from a remote machine (e.g. my laptop) via CLI commands and still be able to use existing conda environments via the Nebari Workflow Controller?

Aside: Although I couldn't figure out how to run Argo workflows on the Nebari k8s from my laptop, I did manage to launch jobs there using SkyPilot. I couldn't take advantage of any of the Nebari goodies like conda-store environments of course.

[dharhas]

I'm pretty sure I have done this a while back. If you setup credentials correctly there shouldn't be anything stopping you. I'll try and look up my old notes.

The conda-store env availability is of course the biggest usability issue. There is a way to make the conda env available via argo workflow controller (https://www.nebari.dev/docs/how-tos/setup-argo/) but it is not well documented on how to use it right now. @kcpevey was doing some experiments with Hera and was struggling with activating environments etc.

[kcpevey]

The outcome of my struggles with argo/hera is summarized here in the docs. I ended up creating some helper functions (included) which help you construct a single line command to execute a python script in a conda environment via the argo scheduler.

[kcpevey]

It sounds like the argo cli isn't using that ARGO_TOKEN. Have you tried to feed the token directly to the argo cli to authenticate? I've only used the cli a few times a while back so I don't know much about it.

[Adam-D-Lewis]

@maxrjones I looked into this a bit and found that The ARGO_TOKEN set by default works if you modify it to add Bearer (Bearer then a space, also it's case-sensitive) before what it is currently set to. That is a bug.

[Adam-D-Lewis]

Looking into it a bit more, it's actually that way b/c hera (python client for argo workflows) expects the token without "Bearer " and it was designed for that use case. I'll propose that we set a different env var HERA_TOKEN for use with Hera in an issue.

[Adam-D-Lewis]

Opened an issue: #2437

[maxrjones]

Thank you @Adam-D-Lewis for figuring that out! Updating ARGO_TOKEN to include Bearer would make using argo via Nebari so much more convenient for us, so very excited about #2438 🎉