URL schema - include ids or not?

[vsoch]

We currently have two patterns of urls - the first does not include any identifier in the URL, e.g.,

`POST /m1/workflow/update/`

But then provides it in the request

{"id": 123...}

The latter has the identifier in the url directly:

`GET /m1/workflow/<workflow id>/`

I typically do the latter, and it looks like most urls are already conforming to this, so it would come down to updating the update endpoint to do the same, e.g,

`POST /m1/workflow/<workflow-id>/update/`

Would this work @fgypas ?

[vsoch]

Hmm actually it looks like the update function is part of the app (not the api?)

@app.route('/update_workflow_status', methods=['POST'])
def update_status():
    update_form = SnakemakeUpdateForm()
    errors = update_form.validate(request.form)

    if errors:
        abort(404, str(errors))
    else:
        r = update_form.load(request.form)
    # now all required fields exist and are the right type
    maintain_jobs(msg=r["msg"], wf_id=r["id"])
    return "ok"

Is the app view also serving as an API endpoint? And is there a reason to have the id in the form instead of the url? Perhaps we could have a common API endpoint that also serves this form update view? To start I could leave this app function as is, and just add the update function to the API, and then someone familiar with the UI could decide how to possibly refactor the UI.

[fgypas]

@vsoch The story behind comes from the fact that we first developed the /update_workflow_status where we update the pipeline in this unorthodox way and then we realized, that we need an actual API. I think that the id should be part of the url as you recommend. Something I am not sure is what ID should the URL have. The autoincremented by panoptes (https://github.com/panoptes-organization/panoptes/blob/master/panoptes/schema_forms.py#L27) or the hash (https://github.com/panoptes-organization/panoptes/blob/develop/panoptes/app.py#L106) that sends to snakemake? This is also important for snakeface, so let's make a decision that is best for the future, and do not worry about introducing big changes for panoptes. We can handle them.

Does that answer your question, or did I confuse you more?

[vsoch]

That answers my question! And maybe I can help to think about yours.

So the simplest idea is that the server should hold some unique id that can be queried to retrieve the correct workflow. This is assuming that snakemake can hold on to this identifier throughout the life cycle of the job, which it looks like is possible since the logger holds a server id set here. And the id is the first of the two you listed, the database id that auto-increments (here) would be logical. But looking at your hash, could you walk me through what exactly this is? It looks like it's set randomly on init, but based on creating a new workflow, would that not go into the id field? So is your question about whether the id should be auto-incremented or a random uuid? I'd say that is up to the implementation.

The second part to this is whether snakemake would be able to figure out that a rerun of the "same" workflow (that creates another identifier) is already linked to one in the database. I think for panoptes it's not possible to do this? But for snakeface, at least if the workflow is created as a database model first, it could hold multiple WorkflowRuns, each having a different id but being associated with the same workflow. But for what would be sent to snakemake, it would still be the workflow run id.

[fgypas]

So, what would be ideal for me is that users can start a pipeline either from the command line or from the UI (but have the option to restart it from the other).

What I mean is the following (simple use case):

• User starts a pipeline from the web interface
• The pipeline gives an error
• The user fixes the error in the terminal and starts snakemake from the command line
• Ideally, the server should realize that this is the same workflow.

@vsoch I think that this would be possible neither for panoptes, nor for snakeface. Am I right?

[vsoch]

Let's walk through each case:

User starts a pipeline from the web interface

This would mean that a workflow (or workflow run) is created with a unique id, and it gets launched on behalf of the user from the server. Likely what we need here is to be able to pass the workflow id from the server to snakemake, and then snakemake will again ping the server for updates, etc. I think this would actually work, and there would be two cases:

Case 1: no workflow id is provided

As it works now, this would mean that snakemake would request creation of the workflow, getting back the id and storing it to request updates.

Case 2: workflow id is provided

In the case that the workflow is created in the web interface, it would generate it's id then, and then get passed to snakemake. Snakemake would simply skip the step of creating it because it already exists.

So to walk through your example, the user would start the pipeline from the web interface, which would trigger snakemake to run and provide an ID. The pipeline would error, and then the user would need to fix the error, and again provide the monitor flag, and also provide the workflow id.

On the flip side, the user could start the workflow from the command line (without an id) and then have it created in the interface, and assigned an id then.

The one variable that I'd need to think about for my implementation is how to represent workflows vs. running instances. Should it be the case that a single workflow can have more than one run (e.g., a retry or another run) or should we reduce redundancy and simply have a workflow associated with one run, and if it fails, it's essentially reset and restarted?

So I think as long as we are able to provide an identifier to the monitor this should work. What do you think? Should we ask Johannes about adding another variable to specify an id, or extending the --wms-monitor command to be a string that we parse of settings?

[fgypas]

@vsoch Thanks a lot for this. I think some feedback/opinion from @johanneskoester would be great on this. Maybe there is a recommended way (e.g. via the hidden .snakemake files) that we are not aware of.

[johanneskoester]

Let's walk through each case:

User starts a pipeline from the web interface

This would mean that a workflow (or workflow run) is created with a unique id, and it gets launched on behalf of the user from the server. Likely what we need here is to be able to pass the workflow id from the server to snakemake, and then snakemake will again ping the server for updates, etc. I think this would actually work, and there would be two cases:

Case 1: no workflow id is provided

As it works now, this would mean that snakemake would request creation of the workflow, getting back the id and storing it to request updates.

Case 2: workflow id is provided

In the case that the workflow is created in the web interface, it would generate it's id then, and then get passed to snakemake. Snakemake would simply skip the step of creating it because it already exists.

So to walk through your example, the user would start the pipeline from the web interface, which would trigger snakemake to run and provide an ID. The pipeline would error, and then the user would need to fix the error, and again provide the monitor flag, and also provide the workflow id.

On the flip side, the user could start the workflow from the command line (without an id) and then have it created in the interface, and assigned an id then.

The one variable that I'd need to think about for my implementation is how to represent workflows vs. running instances. Should it be the case that a single workflow can have more than one run (e.g., a retry or another run) or should we reduce redundancy and simply have a workflow associated with one run, and if it fails, it's essentially reset and restarted?

So I think as long as we are able to provide an identifier to the monitor this should work. What do you think? Should we ask Johannes about adding another variable to specify an id, or extending the --wms-monitor command to be a string that we parse of settings?

sounds good!
About ids, workflow vs workflow run:
There can indeed be multiple workflow runs spawned from one workflow (+config/workdir) at the same time. E.g. when requesting different targets. Snakemake already takes care of avoiding conflicts for such cases, by its locking mechanism. I think you could savely create a unique id from workdir plus the set of all input and output files, as it is used for the locking mechanism (see here). Just sha-hash them lexigraphically sorted and you should be save.

[fgypas]

Hi @johanneskoester
Thanks a lot for your reply. One question about this. What happens with the checkpoints where you do not have a pre-defined number of input/output files? Would the unique key still work? Or do you mean that we have to care only for the first input and the last output files?