Error in rule spades --No module named 'distutils'-- | Error! The Snakemake workflow aborted.

[Dkyuan]

Hi, akcorut:

Thanks for the pipeline.

After running the pipeline, it aborted with the following errors: 1)Error in rule spades 2)Error! The Snakemake workflow aborted.

==>cat -n run2.log | grep -A 4 -E "Error|error" 
 29434	Error in rule spades:
 29435	    jobid: 1334
 29436	    input: results/fetch_reads_with_kmers/Gly/merged_reads/Gly_reads_with_kmers.merged.sorted.R1.fastq, results/fetch_reads_with_kmers/Gly/merged_reads/Gly_reads_with_kmers.merged.sorted.R2.fastq
 29437	    output: results/assemble_reads_with_kmers/Gly/assembly/contigs.fasta, results/assemble_reads_with_kmers/Gly/assembly/assembly_graph_with_scaffolds.gfa, results/assemble_reads_with_kmers/Gly/assembling_source_reads.done
 29438	    log: logs/assemble_reads/Gly/assemble_reads.spades.log (check log file(s) for error details)
 29439	    conda-env: /disk/disk1/xuan/kmerFlow_451/.snakemake/conda/93fe6492c51dbf0e4f6310c59c861ef8_
 29440	    shell:
 29441	        
 29442	        spades.py --careful --only-assembler -t 8  --pe1-1 results/fetch_reads_with_kmers/Gly/merged_reads/Gly_reads_with_kmers.merged.sorted.R1.fastq --pe1-2 results/fetch_reads_with_kmers/Gly/merged_reads/Gly_reads_with_kmers.merged.sorted.R2.fastq -o results/assemble_reads_with_kmers/Gly/assembly 2> logs/assemble_reads/Gly/assemble_reads.spades.log
----------
 30244	Exiting because a job execution failed. Look above for error message
 30245	Complete log: .snakemake/log/2024-11-07T113757.764885.snakemake.log
 30246	Checking if all required files are provided...
 30247		All required files are present!
 30248	Error! The Snakemake workflow aborted.
 30249	ERROR:root:Error running Snakemake: Command '['snakemake', '--use-conda', '--conda-frontend', 'conda', '--cores', '16', '--snakefile', '/disk/disk2/work/xuan/anaconda3/envs/kgwasflow/lib/python3.11/site-packages/workflow/Snakefile', '--rerun-triggers', 'mtime', 'params', 'input', 'software-env', 'code']' returned non-zero exit status 1.

For the first errr: 1) Error in rule spades
I checked the file "assemble_reads.spades.log" and it indicated "No module named 'distutils'"

==>cat logs/assemble_reads/Gly/assemble_reads.spades.log
/disk/disk1/xuan/kmerFlow_451/.snakemake/conda/93fe6492c51dbf0e4f6310c59c861ef8_/share/spades/spades_pipeline/support.py:508: SyntaxWarning: invalid escape sequence '\d'
  return [atoi(c) for c in re.split("(\d+)", text)]
Traceback (most recent call last):
  File "/disk/disk1/xuan/kmerFlow_451/.snakemake/conda/93fe6492c51dbf0e4f6310c59c861ef8_/bin/spades.py", line 26, in <module>
    import support
  File "/disk/disk1/xuan/kmerFlow_451/.snakemake/conda/93fe6492c51dbf0e4f6310c59c861ef8_/share/spades/spades_pipeline/support.py", line 23, in <module>
    from distutils.version import LooseVersion
ModuleNotFoundError: No module named 'distutils'

After searching the solution for "No module named 'distutils'", it said the the problem might come with new python 3.12, ( mine is 3.11.10 ). Followed the solution:
checking if I had distutils: it showed that I had the module

==>python -m distutils.util
<frozen runpy>:128: RuntimeWarning: 'distutils.util' found in sys.modules after import of package 'distutils', but prior to execution of 'distutils.util'; this may result in unpredictable behaviour

Installing setuptools: it showed that I already have the setuptools.

==>pip install setuptools
Requirement already satisfied: setuptools in /disk/disk2/work/xuan/anaconda3/envs/kgwasflow/lib/python3.11/site-packages (75.3.0)
==>pip install --upgrade setuptools
Requirement already satisfied: setuptools in /disk/disk2/work/xuan/anaconda3/envs/kgwasflow/lib/python3.11/site-packages (75.3.0)

So I don't know how to deal with this problem, would you please kindly give me some suggestions ?

For the second error: may be not a problem after deal with the first error

I have two more questions:
1: Can I start the pipeline at middle steps ? For example, can I just start at where I met the above errors ? and how if yes?
2: How the pipeline extract reads with significant associated k-mers ?
eg:
I found a kmer in kmers/Gly_kmers_list.fa

==>cat fetch_kmers/Gly_kmers_list.fa 
>kmer1_6.37946e-11
ACCCCGAAGCCACCGGCTACCAAGCCGCCCA

then I use grep above-kmer acc1.clean_1.fq.gz; I got just one read sequence
when grep above-kmer fetch_reads_with_kmers/Gly/individual_reads/acc1_reads_with_kmers_R1.fastq, I also got just one read sequence; but there are 5 reads in the file acc1_reads_with_kmers_R1.fastq.
So I was wondering how the pipeline found 4 more reads.

Thanks a lot again !