get_cpus_per_task() assigns 1 to cpus_per_task when not specified

[therealjpetereit]

Hello and merry 2025 🎉🙌

I am using the execellent Snakemake workflow automation our SLURM HPC Cluster in Western Australia.

As most HPC clusters there are specific rules on what job submissions are allowed and which are not.

For us there is a rule that if I request a GPU node I cannot set cpus_per_task=x

The problem I am encountering is that when I omit cpus_per_task=x from my config.yaml or snakefile resources, the slurm executor
plugin will automatically set cpus_per_task=1 as defined by the get_cpus_per_task() function.

Is there a way to instead of setting it to 1, to make slurm do this?

lets say if I do cpus_per_task='default' the plugin would simply omit setting a cpus_per_task value

Cheers
Jakob

[cmeesters]

Thank you for bringing this issue to my attention. Interesting. Can you use --cpus-per-gpu instead?

I will polish the PR on GRES (see snakemake/snakemake-executor-plugin-slurm#173 ) in the coming days and wonder whether it helps to code "if a gpu is selected, set --cpus-per-gpu" instead. Of course, the idea is to provide a solution, that works everywhere. It appears to be a rather futile task. My fellow admins have quite a fantasy when deviating from standards, and SchedMD (the company behind SLURM) occasionally has funny ideas, too.

[therealjpetereit]

Hi,

Can you use --cpus-per-gpu instead?
Depends how it's implemented, the way they use it in the cluster config is that cpus_per_task is red flagged if gpu partition is used. The system has net cpus_per_gpu=8 in our case, but that number is arbitrary and could be anything in the future.

So my recommendation would be to change make cpus_per_task in the snakemake sheduler accept a 'none' argument and it omits the line where cpus_per_task is added to the sbatch command.
I saw that in one of the helper scripts.

I'll track that down an add it as extra comment.

the idea is to provide a solution, that works everywhere.

here's hoping 😂😅

Cheers
J

[therealjpetereit]

Ok,

so I think if you look at lib/python3.12/site-packages/snakemake_executor_plugin_slurm/__init__.py

it has in line 178
call += f" --cpus-per-task={get_cpus_per_task(job)}"

that could have an if not statement for cpus_per_task='none'

Maybe you have a better idea, this is just what came to my mind.

Thanks for the prompt responses 🙌

[cmeesters]

My idea would be to still have a default. 1 CPU is fine for most gpu tasks. So, get_cpus_per_task(job) should be

1. renamed
2. return --cpus-per-gpu=<value> if a gpu is selected
3. else return --cpus-per-taks=<value>

where <value> is equal to threads of a resource section or threads directive of a rule and will take cpus_per_taks or cpus_per_gpu as a fallback.

Does this make sense to you? And would it work on your cluster?

The idea is to keep the workflow itself generic. If 8 is the upper value for now, that is a non-issue. You might be using a multi-gpu + multi-cpu application (in the future), where this proposed layout might come in handy. For know, it will not hurt, or will it?

BTW to me there is no difference in --cpus_per_task or --cpus_per_gpu, because a gpu-task is a process like any other regarding the scheduling process (SLURM is rather ignorant to the offloading to gpus). The difference might be imposed, because otherwise admins need to implement a plugin, to deny cpu-only jobs in gpu partitions. But that is only my take.

[therealjpetereit]

Hi,

I just tried if --cpus_per_gpu works but sadly it doesn't. I assume it still does cpus_per_task in the background.

sbatch --export=ALL -A XXX-gpu' -p gpu -t 600 --mem 1000 --ntasks=1 --cpus-per-task=5 --gpus=1 test.sh
sbatch: error: lua: cli_filter: cannot explicitly request CPU resources for GPU allocation; each allocated GPU allocates 8 cores
sbatch: error: cli_filter plugin terminated with error

sbatch --export=ALL -A 'XXX-gpu' -p gpu -t 600 --mem 1000 --ntasks=1 --cpus-per-gpu=5 --gpus=1 test.sh
sbatch: error: lua: cli_filter: cannot explicitly request CPU resources for GPU allocation; each allocated GPU allocates 8 cores
sbatch: error: cli_filter plugin terminated with error

Your logic is sound, sadly the coding of the cli_filter doesn't allow it in my specific case.
Our cluster basically takes the --gpus=x and then sets the calue for cpus, which collides with setting any cpus.

I did tell teh HPC staff that this is an issue, but alas I haven't heard from them.

[cmeesters]

cannot explicitly request CPU resources for GPU allocation; each allocated GPU allocates 8 cores

ah, this is what you meant. My bad. Still doesn't make sense. Even, if you get assigned n cpus, you should not be able to oversubscribe but explicitly underusing a resource, which otherwise does not get used anyway? (Don't tell me that a signification fraction of GPU applications will use exactly 8 cpus! ;-)

It appears your admin team wrote a filter script. Now, I could set point 2 in my list to simply returning no string, if there is only one cpu selected or none are selected. I do not want users to explicitly write None either way.

[therealjpetereit]

Hi,

yes you are correct, it's a bit of a silly filter.

That solution should work for me, as long maybe return no string if cpus are set to zero ?
Anyway, I am happy with any solution =)

Cheers
J

[therealjpetereit]

Hi again,

I just came across the same situation for memory, once you have a solution for the cpus_per_task, would you mind applying the same to --mem ?

Cheers
J

[cmeesters]

I'm nearing completion. Hopefully, a fix to all our liking will be released next week.

Yet, I do not understand your last comment. We have

if job.resources.get("mem_mb_per_cpu"):
      call += f" --mem-per-cpu {job.resources.mem_mb_per_cpu}"
elif job.resources.get("mem_mb"):
      call += f" --mem {job.resources.mem_mb}"
else:
      self.logger.warning(
          "No job memory information ('mem_mb' or 'mem_mb_per_cpu') is given "
          "- submitting without. This might or might not work on your cluster."
      )

(hopefully, I got the indentation right - see lines 235 et seqq., ought to be correct there.)

Anyway, there is no default setting to be omitted. These parameters are just set if required by a user.

Cheers
Christian

[therealjpetereit]

Hi,

I saw you branched and changed the function.

I tried to install the branch via Github and then install the executor plugin over it, but it didn't work.
I assume there's more to be done ?

Let me know if I can do anything to help.

Cheers
Jakob

[cmeesters]

The PR is still pending. I am sorry, but I fell ill and have a course to teach for the coming two days. I will have a look and try to boost the merger next week. Promised.

You can test the PR (#28) when pulling from this repo and the slurm executor repo (see PR #173 at snakemake/snakemake-executor-plugin-slurm#173).

I hate to leave people in this state - you have work to be done - but cannot help it. Take care.

[therealjpetereit]

Absolutely no problem. Recover and take your time.

I run the test and it will most likely work for me.

Happy to do extra testing if you need any.

[therealjpetereit]

I pulled the #28 and #173 and your changes fix the cpu problem. 🥳🥳🥳

The memory part doesn't seem to accept the 'no string'

if job.resources.get("mem_mb_per_cpu"):
      call += f" --mem-per-cpu {job.resources.mem_mb_per_cpu}"
elif job.resources.get("mem_mb"):
      call += f" --mem {job.resources.mem_mb}"
else:
      self.logger.warning(
          "No job memory information ('mem_mb' or 'mem_mb_per_cpu') is given "
          "- submitting without. This might or might not work on your cluster."
      )

There must be another function that regulates the input. I tried setting mem_mb to some random value in my config.yaml or my resources flag in the snakefile and it accepts that value (I used mem_mb=666 as trial)

host: setonix-01
Building DAG of jobs...
SLURM run ID: 67e91249-84ab-44ae-a6de-9029691987e9
Using shell: /usr/bin/bash
Provided remote nodes: 100
Job stats:
job          count
---------  -------
all              1
run_model        1
total            2

Select jobs to execute...
Execute 1 jobs...

[Thu Jan 30 11:21:49 2025]
rule run_model:
    input: ../data/sv_so_apso_cov2019_nov24.csv, ../data/Covariates_WA_List_with_Source_1April2024.csv
    output: ../reports/SO_COLVCM_best_models_and_meta.xlsx
    log: ../logs/SO_COLVCM_run_model.log
    jobid: 2
    benchmark: ../benchmarks/SO_COLVCM_run_model_benchmark.txt
    reason: Missing output files: ../reports/SO_COLVCM_best_models_and_meta.xlsx
    wildcards: response_col=SO_COLVCM
    resources: mem_mb=666, mem_mib=636, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=XXX-gpu, slurm_partition=gpu, gpus_per_task=1, cpus_per_task=-1, runtime=600, slurm_extra=--gpus=1 --test-only --output=logs_slurm/%j.out --error=logs_slurm/%j.err --nice=10 --gres=gpu:1

WorkflowError:
SLURM sbatch failed. The error message was sbatch: error: lua: cli_filter: cannot explicitly request memory for GPU allocation; each allocated GPU allocates 29440 MB of memory
sbatch: error: cli_filter plugin terminated with error

But when I remove the flag, it goes back to 1000mb as standard.

host: setonix-01
Building DAG of jobs...
SLURM run ID: 1ac8548b-11a3-4088-be8c-60c1fa21e28f
Using shell: /usr/bin/bash
Provided remote nodes: 100
Job stats:
job          count
---------  -------
all              1
run_model        1
total            2

Select jobs to execute...
Execute 1 jobs...

[Thu Jan 30 11:17:28 2025]
rule run_model:
    input: ../data/sv_so_apso_cov2019_nov24.csv, ../data/Covariates_WA_List_with_Source_1April2024.csv
    output: ../reports/SO_COLVCM_best_models_and_meta.xlsx
    log: ../logs/SO_COLVCM_run_model.log
    jobid: 2
    benchmark: ../benchmarks/SO_COLVCM_run_model_benchmark.txt
    reason: Missing output files: ../reports/SO_COLVCM_best_models_and_meta.xlsx
    wildcards: response_col=SO_COLVCM
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=xxx-gpu, slurm_partition=gpu, gpus_per_task=1, cpus_per_task=-1, runtime=600, slurm_extra=--gpus=1 --test-only --output=logs_slurm/%j.out --error=logs_slurm/%j.err --nice=10 --gres=gpu:1

WorkflowError:
SLURM sbatch failed. The error message was sbatch: error: lua: cli_filter: cannot explicitly request memory for GPU allocation; each allocated GPU allocates 29440 MB of memory
sbatch: error: cli_filter plugin terminated with error

There is of course the possibility that I messed up somewhere, but I jsut don't see it.

Any ideas ?

Cheers
J

[therealjpetereit]

Hi again,

I tracked down where the mem argument get's set to 1 if it's deliberately set to 0

resources.py has a function called infer_resources

def infer_resources(name, value, resources: dict):
    """Infer resources from a given one, if possible."""
    from humanfriendly import parse_size, parse_timespan, InvalidTimespan, InvalidSize

    if isinstance(value, str):
        value = value.strip("'\"")

    if (
        (name == "mem" or name == "disk")
        and isinstance(value, str)
        and not isinstance(value, TBDString)
    ):
        inferred_name = f"{name}_mb"
        try:
            in_bytes = parse_size(value)
        except InvalidSize:
            raise WorkflowError(
                f"Cannot parse mem or disk value into size in MB for setting {inferred_name} resource: {value}"
            )
        resources[inferred_name] = max(int(math.ceil(in_bytes / 1e6)), 1)
    elif (
        name == "runtime"
        and isinstance(value, str)
        and not isinstance(value, TBDString)
    ):
        try:
            parsed = max(int(round(parse_timespan(value) / 60)), 1)
        except InvalidTimespan:
            raise WorkflowError(
                f"Cannot parse runtime value into minutes for setting runtime resource: {value}"
            )
        resources["runtime"] = parsed

and that sets the mem_mb to 1 if it's 0. That generally makes sense, but not in SLURM, because in SLURM a mem of 0 assumes all avaiable memory of a node (correct me if I am wrong)

I adjusted that in my local version, but you want to probably change this, so your no memory given function works

Cheers
J