`SLURM sbatch failed` causes Snakemake crash

[nikostr]

Software Versions

snakemake: 8.27.1
snakemake-executor-plugin: 1.3.6
SLURM: 24.05.4

Describe the bug
sbatch: error: Batch job submission failed: Requested node configuration is not available leads to Snakemake crashing with WorkflowError, with jobs being left running/in the SLURM queue.

A minimal improvement would be for the workflow to shut down nicely, either canceling jobs and cleaning up after them, or waiting for them to finish. Another option would be to treat this like other job failures, allowing to workflow to keep going, and even allowing retries for jobs failing due to this. Also, specifying which exact call to SLURM caused the error would help with troubleshooting these issues.

Logs

The full error message:

WorkflowError:
SLURM sbatch failed. The error message was sbatch: error: Batch job submission failed: Requested node configuration is not available

Additional context
My cluster's resource limits aren't super clear. Some combination of resources generated in later attempts are clearly not okay.

[cmeesters]

Thank you for your issue report. Please note:

• the current Snakemake version is 9.5.1 . Yours is a bit outdated. During the Snakemake Hackathon in March we made a number of fixes potentially impacting cluster execution, too.
• However, your particular error indicates, that you requested something, which your cluster does not provide. Please check with snakemake [<args>] --verbose. There you see the actual sbatch call made by the SLURM executor.

[nikostr]

Sorry if I was unclear :) I understand where the error comes from. I have now attempted running this with Snakemake v9.5.1 and the problem persists.

My issue is not the error itself, but how it's handled. Snakemake crashes without cleanup, both pending and running jobs are left hanging in the queue. At least to me it's non-obvious what happens to these jobs. How will snakemake handle the output on restart? If one of the jobs errors out, will snakemake discard any incomplete results or do I risk having incomplete results affect downstream steps?

Regarding --verbose: I am not always interested in every sbatch call, but I am interested in failing sbatch calls. Reading the code it looks like adding this info to the error message would be straight forward, e.g. by inserting a {call} in the error string here:

snakemake-executor-plugin-slurm/snakemake_executor_plugin_slurm/__init__.py

Line 303 in ec82a70

f"SLURM sbatch failed. The error message was {e.output}"

Here is a small Snakefile and workflow profile that reproduces the issue:

Snakefile:

rule all:
    input:
        "results/a",
        "results/b",
        "results/c",


rule gen_a:
    output:
        "results/a",
    shell:
        "sleep 5 && touch {output}"


rule gen_b:
    output:
        "results/b",
    shell:
        "sleep 50 && touch {output}"


rule gen_c:
    input:
        rules.gen_a.output,
    output:
        "results/c",
    shell:
        "touch {output}"

Profile (change the resources of gen_c to something that produces the appropriate error):

executor: slurm
jobs: 2
keep-going: true
printshellcmds: true
default-resources:
  runtime: "3m"
  slurm_partition: shared
set-resources:
  gen_c:
    mem: '200GB'

[cmeesters]

I'm sorry. I noticed that I cannot concentrate well, today.

Anyway, the immediate crash happens consistently? Indeed, any error should lead to a shutdown of Snakemake and a subsequent cleanup. In this case, the workflow ought to keep running for jobs running and pending.

Please update your Snakemake version before trying to confirm.

[nikostr]

No worries, we all have those days :) To be clear, this isn't urgent :)

I have already updated my snakemake version.

I'm not sure that I'm following? When an sbatch call fails a WorkflowError is raised, if I understand the following code correctly:

snakemake-executor-plugin-slurm/snakemake_executor_plugin_slurm/__init__.py

Lines 288 to 304 in ec82a70

	try:
	process = subprocess.Popen(
	call,
	shell=True,
	text=True,
	stdout=subprocess.PIPE,
	stderr=subprocess.PIPE,
	)
	out, err = process.communicate()
	if process.returncode != 0:
	raise subprocess.CalledProcessError(
	process.returncode, call, output=err
	)
	except subprocess.CalledProcessError as e:
	raise WorkflowError(
	f"SLURM sbatch failed. The error message was {e.output}"
	)

Since this leads to the workflow immediately crashing when I run this with the latest version of snakemake I'm assuming that this isn't being caught and handled anywhere, but I may very well misunderstand something.

[nikostr]

Oh, and regarding the Snakefile above: you might need to tweak the sleep times depending on the queues etc. For me these values worked. What needs to happen is that gen_b needs to be pending or running when gen_c is submitted and crashes, leaving gen_b in the queue without it being handled by snakemake.

[nikostr]

This seems to be how KeyboardInterrupt is handled in the snakemake scheduler:
https://github.com/snakemake/snakemake/blob/960f6a89eaa31da6014e810dfcf08f635ac03a6e/src/snakemake/scheduler.py#L343-L347

Maybe something similar could be used in the below piece of code? I.e., instead of raising a WorkflowError it could either run something like self.cancel() to provide a nicer clean up or perhaps self.report_job_error(job) to mark only this job as having failed?

snakemake-executor-plugin-slurm/snakemake_executor_plugin_slurm/__init__.py

Lines 301 to 304 in ec82a70

	except subprocess.CalledProcessError as e:
	raise WorkflowError(
	f"SLURM sbatch failed. The error message was {e.output}"
	)

Maybe these solutions are way too naive. I'm not familiar with this codebase, and I'm really sorry if these proposals are nonsense.

[cmeesters]

A workflow error gets propagated such that the executor will be cancelled. The two different except statements in the executor are due to different exceptions. One, where we have a specific error. One, where we see something as an error message in stderr, but a return code of 0. Weird, but this can happen with SLURM.

So far, I have seen immediate crashes myself. Yet, not consistently. Always in association with cluster issues. So, not sure, what's going on. Can you attach a log with --verbose.

[nikostr]

Yes, snakemake crashes immediately, but as it crashes it does not handle any of the jobs in the SLURM queue. It does not cancel jobs that are running or pending, nor does it wait for these to finish before shutting down. I'm attaching the verbose log. Again, I understand why the error occurs, but I think that it should be handled differently, either by simply reporting the job as failed, or cleaning up the SLURM queue before exiting.

2025-06-08T222122.762049.snakemake.log

[nikostr]

I ended up writing a PR with a potential solution. I'm not sure if it's any good, feel free to have a look: #322