Parsing issue when using gres

[sprdmt]

Software Versions
snakemake version 9.6.0
snakemake-executor-plugin-slurm version 1.4.0
slurm 25.05.0

Describe the bug
Parsing problem occurs when I am trying to submit the snakemake rules with gres specified. I tried both gres="shard:1" and gres="gpu:1" (since our cluster supports GPU shards). When gres is defined, the job gets sent correctly to a GPU node, however, it immediately crashes with an error message:

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided remote nodes: 1
Provided resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, cpus_per_task=1, jobs=20, tasks=-1
Select jobs to execute...
Execute 1 jobs...

[Sun Jun 22 18:15:32 2025]
rule test_gpu:
    output: gpu_test/.done
    jobid: 0
    reason: Forced execution
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=***, slurm_user=***, cpus_per_task=1, runtime=1, jobs=20, gres=gpu:1, slurm_partition=gpu, tasks=-1
Shell command: 
        ml conda
        conda activate semibin-2.2.0
        mkdir -p gpu_test
        echo "Running on GPU node " > gpu_test/test_output.txt
        echo "GPU test job completed." >> gpu_test/test_output.txt
        SemiBin2 single_easy_bin -h >> gpu_test/test_output.txt 
        nvidia-smi >> gpu_test/test_output.txt
        touch gpu_test/.done
        
[2025-06-22T18:15:32.749] error: TaskProlog failed status=230
PARSE ERROR: Argument: 496
             Couldn't find match for argument

Usage: 
   dcgmi group --host <IP/FQDN> -l -j 
   dcgmi group --host <IP/FQDN> -c <groupName> --default
        --defaultnvswitches 
   dcgmi group --host <IP/FQDN> -c <groupName> -a <entityId> 
   dcgmi group --host <IP/FQDN> -d <groupId> 
   dcgmi group --host <IP/FQDN> -g <groupId> -i -j 
   dcgmi group --host <IP/FQDN> -g <groupId> -a <entityId> 
   dcgmi group --host <IP/FQDN> -g <groupId> -r <entityId> 


For complete USAGE and HELP type: 
   dcgmi group --help`

Minimal example

I use the command
snakemake --snakefile Snakefile --profile slurm-lisc --jobs 1

my glogal config.yaml is:
executor: slurm
jobname: "{rulename}.{jobid}.sh"
retries: 0
latency-wait: 60
keep-going: true
use-conda: false
printshellcmds: true

default-resources:
mem_mb: 1000
slurm_account: ''
slurm_user: ''
cpus_per_task: 1
runtime: 1h
jobs: 20

My slurm-profile.sh is:
#!/bin/bash
#SBATCH --job-name=smk-{rule}
#SBATCH --account={resources.slurm_account}
#SBATCH --output=logs/{rule}.{jobid}.out
#SBATCH --error=logs/{rule}.{jobid}.err
#SBATCH --mem={resources.mem_mb}M
#SBATCH --time={resources.runtime}
#SBATCH --mail-user=***
#SBATCH --mail-type=ALL
#SBATCH --no-requeue
#SBATCH --cpus-per-task={threads}

ml conda

{exec_job}

My Snakefile is:
rule test_gpu:
output:
"gpu_test/.done"
threads: 1
resources:
mem_mb=1000,
gres='shard:1',
runtime=1
shell:
"""
ml conda
conda activate semibin-2.2.0
mkdir -p gpu_test
echo "Running on GPU node " > gpu_test/test_output.txt
echo "GPU test job completed." >> gpu_test/test_output.txt
SemiBin2 single_easy_bin -h >> gpu_test/test_output.txt
nvidia-smi >> gpu_test/test_output.txt
touch {output}
"""

Additional context

if I run the same script but do not specify gres, it runs perfectly fine on a CPU node. Seems like the problem is with parsing the gres value but I lack the knowledge to solve it myself.
Thanks in advance!

[cmeesters]

I'm afraid, you are

• apparently mixing this plugin with the old SLURM profile and
• encountering a task prolog error.

Please tell us, whether you are actually using this plugin - and where this profile comes in (the shown command line does not work like expected). You should not be using it at all in combination with this plugin.

Also, you might need to ask your admins to know what the prolog error actually is. For, this is cluster related. Your job might have provoked it. But that is rather unusual.

[sprdmt]

Dear Christian,
what do you mean by the old SLURM profile?
I shouldn't be using general profiles/my profile is incorrectly configured?

Or that if I use the plugin, I shouldn't be using the profile whatsoever?

I thought the issue is with the prolog script and already contacted our administrator, however, he said the Snakemake installation is standard and I should look for the issue on my end

---

P.S. I think I know what you mean. Yes, my command doesn't refer to the slurm-profile.sh, however, I tried I think all possible options, including specifying the SLURM template manually but the outcome was always the same. I guess it's the Prolog error than and I need to talk to the administrator again. Could you please elaborate on what exactly I should be asking?

[cmeesters]

1. get rid of the Conda and module load statement - there is a Conda directive in Snakemake. With --sdm conda you ensure that Conda applications are autoloaded. There are just too many statements in that shell part of your rule to figure out, what is going on.
2. do not use that profile of yours. A workflow-profile with a yaml definition to hold resources? Yes. A shell script? No.
3. dcgmi is a basic diagnostics tool by NVIDIA. I assume this is called in the task prolog on your cluster. If the error persists (you might want to run Snakemake in verbose mode (--verbose)), perhaps you can interpret the error message yourself. Else approach your admins about it. I have no idea what argument "496" is.

However, note that Snakemake submits itself. Not the cuda application.

edit: PS a Conda module might require a writeable systemwide directory, which might not exist. In this case, Conda is either pointed to a different prefix or you run into issues.

[cmeesters]

what do you mean by the old SLURM profile?

That SBATCH shell script you provide. It is outdated since Snakemake version 8. See https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html