Prevent simultaneous task failures multiple archive calls

[arhall0]

If multiple tasks fail simultaneously, they will all attempt to call beeflow.wf_manager.resources.wf_update.archive_workflow. This will succeed for the first task that makes the call, but then lead to an endless loop of calls for the other tasks that error since the workflow folder will no longer exist in .beeflow/workflows.

This solves that problem by checking the workflow state does not begin with Archived when attempting to archive a failed workflow. db.workflows.get_workflow_state seems to be the only way to do this once a workflow has been archived.

I put this check where a failed task would call archive_fail_workflow, but potentially this could be in archive_workflow directly.

Needs a test before removing WIP.

[arhall0]

I adjusted the solution based on discussion. Since the behavior of a failing task failing and archiving the entire workflow may change, I added a more general check to archive_workflow that prevents archiving if the workflow is already in an Archived state and logs a warning about this. I think this is a more robust fix as it should not need to be changed if task failing behavior changes.

[pagrubel]

Nice I tried a workflow of cat-grep-tar with a 2nd branch starting with running the cat step without a proper input file and all looks good. One branch fails and the workflow is still archived. Here is the dag. I think I may put the workflow in our data directory. I'll add a readme for it.

[pagrubel]

So the workflow that caused the error doesn't cause the error anymore, however, the above workflow continued but the final status was not in the archive so because a task failed on one branch the other branch continued to run but the workflow was archived too early. I've added the example. This workflow should be archived and removed from the ~/.beeflow/workflows directory. It was archived early and the status is incorrect. If you do a beeflow query on it, it looks fine because it is accessing the ~/.beeflow/workflow directory.