option to set the log directory?

[jbloom]

Is it possible to enable the log directory to be set to something other than a directory in .snakemake? For large workflows that submit many jobs, it would be nice to be able to write these to a scratch directory rather than putting tens of thousands of files in .snakemake.

[bentyeh]

I'd like to second the above request and consolidate several previous requests across multiple issues for giving the user more flexibility in specifying how to log stdout and stderr from rules submitted as SLURM jobs:

1. Specify log directory (this issue, option to set the log directory? #94)
2. Specify which wildcards to use, not just concatenate all of them (Interpolation of wildcards in profile slurm configuration #11, add human job name with wildcards #34, SLURM logs path are unreasonably long and contains all wildcards #84)
3. Specify separate names and paths for stdout (to pass to sbatch --output=...) and stderr (to pass to sbatch --error=...) (Separately logging stdout and stderr #95)

More generally, I hope that this new executor framework can at least achieve feature parity with the previous version (e.g., v7) of Snakemake that supported a very flexible logging setup via snakemake --cluster-config cluster.yaml ... <Snakefile>

where cluster.yaml looked something like

__default__:
    time: 01:00:00
    threads: 2
    mem: 10G
    error: "logs/{rule}.{wildcards.sample}.err"
    output: "logs/{rule}.{wildcards.sample}.out"

[cmeesters]

@jbloom no, however, refactoring the logging feature (particularly, deleting log files older than n days) is on my TODO list. The log directory is a core feature - I will discuss this internally. We will work on this, it might take a little while.

@bentyeh please do not mix issues. I ensure, that we take all feature requests seriously. ad 2) I can see your point, however, this too needs some discussion. After all, implementing it the way it is, was a feature request. And a change is not as straightforward as it seems: configurability must be maintained at three points: the command line, the Snakefiles and the profile. Adding a keep-those-wildcadrs-option for the CLI would be awkward, right?

Regarding your other critics: the workflow profile offers just that, albeit that I too would like to see something changed. It ought to be possible to specify a single file, not a directory.

[jbloom]

@cmeesters, great, thank you so much for considering customizability of the log directory as a feature. That would be tremendously useful and I appreciate your time and consideration of this.

[mbhall88]

Agreed, this would be very useful.

A second thing I want to ask for is documenting where the logs are actually output to in the plugin docs (unless I've missed it).

[cmeesters]

it's on my todo list

[blaiseli]

I agree with @mbhall88 that it would be nice if it was easier to find in the plugin documentation how slurm log paths are built. I was almost about to look into the source code to figure out why some of my rules had a weird very long path to their slurm log.
It turned out that, because I wanted to make those rules more generic, they had a {prefix} wildcard, as follows:

rule annotate_with_snpEFF:
    input:
        vcf_in="{prefix}.vcf.gz",
    output:
        # /!\ wildcard prefix gets in the slurm log path too
        # -> the log is burried deep in the .snakemake/slurm_logs dir
        vcf="{prefix}.snpEFF.vcf.gz"