Allow workflows/cronworkflows to be resubmitted

[iameskild]

Currently there is a known limitation which prevents us from resubmitting workflows/cronworkflows. As such, this is preventing the Argo-Jupyter-Scheduler from being able to Resume paused cron-workflows and prevents cron-workflows from being updated (ie to run on a new schedule).

[dharhas]

@iameskild When you have a moment, can you add details or point to an upstream issue that defines why this is? i.e. is this a limitation of Argo or an implementation issue etc.

[iameskild]

@dharhas as far as a I can tell, this is a limitation of the Nebari-Workflow-Controller ( https://github.com/nebari-dev/nebari-workflow-controller#known-limitations ). Resubmitting workflows fail in general when using NWC.

[dharhas]

Yes. But there are no details on why.

[iameskild]

I'm not sure why this is a limitation on the NWC side. @Adam-D-Lewis if you have more information, do you mind chiming in?

[Adam-D-Lewis]

Originally, I didn't support resubmission because I wasn't sure how to get the username from the creator-preferred-username label, but I think Eskild added a way to get the username from the creator-preferred-username, but I need to check that his solution applies here.

[Adam-D-Lewis]

So I checked and unfortunately, the creator-preferred-username set by argo just strips off unsupported characters like @ and ! in the username so getting the creator on a resubmitted workflow is not possible.

It looks like there is a fix that was just committed 2 weeks ago and might be in the latest version of Argo Workflows if we want to upgrade (It should be in the v3.5 release of argo worfklows). With the fix we could just look at the creator label instead of the creator-preferred-username.
argoproj/argo-workflows#11414

[ericdatakelly]

I'm working on the doc for jupyter-scheduler and followed the warning to this issue. I see that argo-workflows is up to v3.5.6. Will this be fixed soon or should I leave the warning unmodified? https://www.nebari.dev/docs/tutorials/jupyter-scheduler#submit-jupyter-notebooks-to-run-on-a-schedule

[Adam-D-Lewis]

@ericdatakelly Updating Argo workflows is desired, but not scheduled at the moment. We would accept a PR doing so, but we would need to update Jupyter scheduler and argo-workflow-controller as well before merging.