Default to use `--container` param in job submission

[Domomod]

Is your feature request related to a problem? Please describe.
I'm facing an issue where singularity/apptainer is not available on login node. Instead my friendly admins aked me to use --container param for sbatch to specify the container.

Describe the solution you'd like
Since --container is a built in of slurm, I would actually expect the plugin to use this when container is specified in rules, and executor is set to slurm.

Describe alternatives you've considered
Another alternative would be to have this behaviour as a optional flag introduced by the plugin.

[cmeesters]

That might be feasible, yet doesn't --sdm apptainer from Snakemake work for you?

Also, if this would be a flag, it would apply globally. It is, however, a Snakemake directive per rule, already. Feel free to elaborate, if I misunderstood you!

[Domomod]

Unfortunatelly in my case apptainer is only available on compute nodes - I am not sure why. I personally think that using --container flag would be more reproductible across diffrent hpc servers.

A qucick fix for me was to use the general-executor-plugin and passing the container to the execution as follow:

executor: cluster-generic
cluster-generic-submit-cmd:
sbatch
--cpus-per-task={threads}
--mem={resources.mem_mb}
--job-name=smk-{rule}-{wildcards}
--output={log.slurm}
--error={log.slurm}.err
--parsable
--container {resources.container}

The downside is I need to redefine the container in resources, because i couldn't figure out how to reference the value from snakemake pipeline definition.

[cmeesters]

@Domomod please apologize: I have written a comment, but apparently did not send it.

So, once more:

in my case apptainer is only available on compute nodes

You might want to find the reason: This does not make sense to me. How is apptainer provided (as a module file? via conda?)? To execute the application, you only need to have it on compute nodes. Yet, to test parameters it apptainer should be available anywhere - on login nodes, too.

Be this as it may, are you using containers in every job? Or just specific jobs? If this is a resource per job and not a global catch-all container, why does

rule:
    ...
    container:
        "<URI or path>"
    ...

not work for you? May we see a workflow definition, parameterization via configs/profiles/the command line and resulting the resulting log?

[Domomod]

Hi so apptainer is available once I connect to the compute node. But it's not installed in login node. It just is installed on the login node. I dont need to run conda nor load a module, and there is no ready module / conda to load the apptainer on login node.

I use specific containers for specific jobs - I define them as you have shared in your example.
Running snakemake on my login node fails, because at some point snakemake tries to run apptainer on the login node.

I believe using --container to specify the container for srun would be more fallproof, as it's a standard part of the srun command.
https://slurm.schedmd.com/containers.html

[cmeesters]

Ok, then let's do it! Do you have a minimal example plus test container?