Possible bug with slurm and storage fs plugin for local keyword

[raphaelbetschart]

I have some issues understanding the local keyword when using the storage fs plugin. I am trying to create a rule which takes two files as input: One input file which should get copied to a compute node and another input file which should NOT get copied.

My Snakefile looks like this:

BAM                = "/storage/{sample}.bam"
CRAM_REMOTE = "/storage/{sample}.remote.cram"
CRAM_LOCAL = "/storage/{sample}.local.cram"

REFERENCE_GENOME = "/storage/reference/hg38.fa"

SAMPLES = ["S1", "S2"]

rule all:
        input:
                expand(CRAM_LOCAL, sample = SAMPLES),
                expand(CRAM_REMOTE, sample = SAMPLES)

rule Remote:
        input:
                BAM = BAM,
                REF = REF
        output:
                CRAM_LOCAL
        envmodules:
                "samtools/1.18"
        threads:
                16
        shell:
                """
                samtools view -@ {threads} -C -T {input.REF} -o {output} {input.BAM}
                """

use rule Remote as Local with:
        input:
                BAM = BAM,
                REF = local(REF)
        output:
                CRAM_REMOTE

In rule Remote I would like to copy both input files to the compute node directory, specified like this remote-job-local-storage-prefix: /scratch/$SLURM_JOB_ID".

In rule Local only the BAM input file should get copied to the compute node.

What happens now when I submit these 4 jobs is that only jobs spawned by rule Remote are getting submitted to SLURM, the two jobs from Local are run locally.

My profile is the following:

default-resources:
  cpus_per_task: 16
  mem_mb: 32000
executor: slurm
default-storage-provider: fs
shared-fs-usage:
  - persistence
  - software-deployment
  - sources
  - source-cache
local-storage-prefix: "/scratch/"
remote-job-local-storage-prefix: "/scratch/$SLURM_JOB_ID"

What I am doing wrong here?

Thanks for any hints.

[johanneskoester]

To be honest, I don't see how it could happen that the rule Local runs locally and is not submitted to slurm here. The only criteria for rules to be not submitted if a non-local executor is selected is when they don't have any output or when they are explicitly marked as local rules. Are you sure that you did not specify localrules anywhere in your workflow (https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#local-rules)? Also, may I ask you to post your Snakemake log here?