Display slurm job id in snakemake log

[lczech]

Is your feature request related to a problem? Please describe.
When jobs fail, I am seeing the following in the Snakemake log (with Snakemake 9.3.3):

This unfortunately does not allow me to easily find the slurm log for this job, making it hard to track down which jobs failed for which reasons.

Describe the solution you'd like
I'd like the slurm job id to be reported in the error message. I think this used to be the case (maybe that was with the old "cluster profile" though), but not any more.

[cmeesters]

Thank you for reporting this issue. Though, I am not sure there is an issue:

Usually, when running with the SLURM executor plugin, your terminal looks like this:

Error in rule sam_to_bam:
    message: SLURM-job '824877' failed, SLURM status is: 'TIMEOUT'. For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 38
    input: alignments/m18_bc06.sam
    output: alignments/m18_bc06.bam
    log: logs/samtools/samtobam_m18_bc06.log, /home/meesters/.snakemake/slurm_logs/rule_sam_to_bam/m18_bc06/824877.log (check log file(s) for error details)
    conda-env: /lustre/project/nhr-zdvhpc/meesters/sdm_conda/cc33a823d07611f7c6cef9b22342f33a_
    external_jobid: 824877
[Wed Jun 18 12:59:14 2025]
Error in rule sam_to_bam:
    message: SLURM-job '824886' failed, SLURM status is: 'TIMEOUT'. For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 42
    input: alignments/ma_bc07.sam
    output: alignments/ma_bc07.bam
    log: logs/samtools/samtobam_ma_bc07.log, /home/meesters/.snakemake/slurm_logs/rule_sam_to_bam/ma_bc07/824886.log (check log file(s) for error details)
    conda-env: /lustre/project/nhr-zdvhpc/meesters/sdm_conda/cc33a823d07611f7c6cef9b22342f33a_
    external_jobid: 824886

Here, Snakemake states what is doing and the executor plugin happily reports that it submitted a job and where to find the logs (configurably with slurm-logdir), thereafter. In case of an error, I see an error line like in the example above:

 log: logs/samtools/samtobam_ma_bc07.log, /home/meesters/.snakemake/slurm_logs/rule_sam_to_bam/ma_bc07/824886.log

It provides me with the Snakemake log and the SLURM log.

So, I wonder: How is Snakemake configured in your case? Are you actually using this plugin?

[lczech]

Hi Christian,

thanks for the reply!

I'm (now) using Snakemake 9.6.2, and calling it like this

snakemake --workflow-profile [...]/workflow/profiles/slurm [...]

(leaving out other options for simplicity)

with workflow/profiles/slurm/config.yaml containing

executor: slurm

(among other stuff).

Also, my jobs are submitted to slurm, and running on the cluster, so that part is working.

My typical start of a rule looks like this

[Tue Jun 24 20:17:46 2025] Group job mapping (jobs in lexicogr. order):
    [Tue Jun 24 20:17:46 2025]
    rule map_reads:
        input: trimming/CRR493616-1.1.fastq.gz, trimming/CRR493616-1.2.fastq.gz, trimming/CRR493616-1.1.fastq.gz.done, trimming/CRR493616-1>
        output: mapping/sorted/CRR493616-1.bam, mapping/sorted/CRR493616-1.bam.done
        log: logs/mapping/bwa-mem2/CRR493616-1.log
        jobid: 75
        benchmark: benchmarks/mapping/bwa-mem2/CRR493616-1.log
        reason: Missing output files: mapping/sorted/CRR493616-1.bam
        wildcards: sample=CRR493616, unit=1
        threads: 8
        resources: mem_mb=57870, mem_mib=55190, disk_mb=15011, disk_mib=14316, tmpdir=<TBD>, slurm_account=caeg, slurm_partition=compregula>
Shell command: None
Job 7ead9980-9987-503c-a92a-c42172a46651 has been submitted with SLURM jobid 8642519 (log: /maps/projects/prohaska/scratch/grapevine/callin>

which looks different from yours, as it contains the slurm jobid in an extra line at the end, instead of in a message line within the rule.

When a rule fails, my output then looks like this

Error in group call_variants
    [Wed Jun 25 07:29:04 2025]
    Error in rule call_variants:
        message: None
        jobid: 5145
        input: mapping/dedup/CRR495179.bam, mapping/dedup/CRR495179.bam.bai, mapping/dedup/CRR495179.bam.done, /maps/projects/prohaska/data>
        output: calling/called/CRR495179-CP126653.1.g.vcf.gz, calling/called/CRR495179-CP126653.1.g.vcf.gz.tbi, calling/called/CRR495179-CP>
        log: logs/calling/gatk-haplotypecaller/CRR495179-CP126653.1.log (check log file(s) for error details)
        conda-env: /maps/projects/willerslev/people/pxl240/software/miniconda-envs/bc332db142a493791dc3c8ffe7fe60ce_

with no hint towards the slurm jobid, and the suspicious line message: None in there.

The screenshot in my original message above however does not have the job id, and was taken from the terminal output, and not from a log file. I'm wondering if this has to do with this issue about log messages not always showing up properly?

However, looking specifically now for further hints, I found that the output in .snakemake/log/ does contain the message hint in some cases, so I'll need to have a closer look - not sure if it's always there.

Also, I am getting a bit confused now, as there is also the per-job .snakemake/slurm_logs/ directory, with the per-job logs. Those usually list the errors, but somehow I do not always find the log files for any given jobid there. That is, find .snakemake/slurm_logs/ -name "*8642519*" for instance (for the above run) does not find the log file. Are those files deleted by snakemake at some point?

Anyway, hope that the first part here clarifies what I am doing, and the second means that I'll need to do some more digging :-)

Happy to hear your thoughts, and so long
Lucas

[cmeesters]

The part I posted was merely the error part. That the SLURM logfile is indicated after submission works as intended. (This is the executor printing the message). Yet, in an error case, you should see both logs. It is rather weird, that you do not see that.

The executor deletes all SLURM logs as Snakemake submits itself and the SLURM logs only contain the same content as the Snakemake logs, just for individual rules, only. Only in an error case, the logs are kept (and deleted after 10 days, if working in one SLURM logfile output directory). Otherwise, we would keep a zillion of logfiles, which will not make your admins happy, because of all the inodes this takes up. You can tweak that behaviour, too. For this, I refer to the documentation.

As for your actual question: I am not sure what is going on. You linked to an issue where you have been creating your own logger instance. That might interfere. I am, however, not able to test anything right now, for lack of a cluster (broken cooling circuits, hopefully fixed, soon).

[cademirch]

I can't seem to reproduce this, unfortunately.

Versions:

❯ pixi list | grep snakemake
snakemake                                9.6.2        hdfd78af_0            9.2 KiB    conda  https://conda.anaconda.org/bioconda/
snakemake-executor-plugin-slurm          1.4.0        pyhdfd78af_0          19.8 KiB   conda  https://conda.anaconda.org/bioconda/
snakemake-executor-plugin-slurm-jobstep  0.3.0        pyhdfd78af_0          12.6 KiB   conda  https://conda.anaconda.org/bioconda/
snakemake-interface-common               1.19.4       pyhdfd78af_0          19.5 KiB   conda  https://conda.anaconda.org/bioconda/
snakemake-interface-executor-plugins     9.3.7        pyhdfd78af_0          22.3 KiB   conda  https://conda.anaconda.org/bioconda/
snakemake-interface-logger-plugins       1.2.3        pyhdfd78af_0          12.2 KiB   conda  https://conda.anaconda.org/bioconda/
snakemake-interface-report-plugins       1.1.0        pyhdfd78af_0          13 KiB     conda  https://conda.anaconda.org/bioconda/
snakemake-interface-storage-plugins      4.2.1        pyhdfd78af_0          19.5 KiB   conda  https://conda.anaconda.org/bioconda/
snakemake-minimal                        9.6.2        pyhdfd78af_0          855.1 KiB  conda  https://conda.anaconda.org/bioconda/

Snakefile:

rule all:
    input:
        "output.txt",


rule process_data:
    output:
        "output.txt",
    log:
        "logs/process_data.log",
    shell:
        "exit 1"

Command: snakemake --executor slurm -j1

Output:

❯ snakemake --executor slurm -j1
host: mustard
Building DAG of jobs...
SLURM run ID: ced7753b-fb32-4fd5-ae7e-9ea3120bfeb4
Using shell: /usr/bin/bash
Provided remote nodes: 1
Job stats:
job             count
------------  -------
all                 1
process_data        1
total               2

Select jobs to execute...
Execute 1 jobs...

[Sat Jun 28 11:45:29 2025]
rule process_data:
    output: output.txt
    log: logs/process_data.log
    jobid: 1
    reason: Missing output files: output.txt
    resources: tmpdir=<TBD>
No SLURM account given, trying to guess.
Unable to guess SLURM account. Trying to proceed without.
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
No job memory information ('mem_mb' or 'mem_mb_per_cpu') is given - submitting without. This might or might not work on your cluster.
Job 1 has been submitted with SLURM jobid 9643915 (log: /private/home/cmirchan/slurmlogtest/.snakemake/slurm_logs/rule_process_data/9643915.log).
[Sat Jun 28 11:46:09 2025]
Error in rule process_data:
    message: SLURM-job '9643915' failed, SLURM status is: 'FAILED'. For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 1
    output: output.txt
    log: logs/process_data.log, /private/home/cmirchan/slurmlogtest/.snakemake/slurm_logs/rule_process_data/9643915.log (check log file(s) for error details)
    shell:
        exit 1
        (command exited with non-zero exit code)
    external_jobid: 9643915
Shutting down, this might take some time.
Exiting because a job execution failed. Look below for error messages
[Sat Jun 28 11:46:29 2025]
Error in rule process_data:
    message: None
    jobid: 1
    output: output.txt
    log: logs/process_data.log (check log file(s) for error details)
    shell:
        exit 1
        (command exited with non-zero exit code)
Complete log(s): /private/home/cmirchan/slurmlogtest/.snakemake/log/2025-06-28T114529.038987.snakemake.log
WorkflowError:
At least one job did not complete successfully.

Can you share your map_reads rule? The message: None is odd, but I have that in my output as well.