MultiQC: Process `MULTIQC` input file name collision #4969

jamesboot · 2024-05-03T13:23:18Z

jamesboot
May 3, 2024

Hi,

I'm new to NextFlow so forgive me if I've missed something very basic here. I'm trying to build a pipeline with the following steps:

Subsample FASTQs to 1M reads (a bit unnecessary but the feature has been requested!)
Perform FASTQC on 1M subsampled FASTQ files
Perform quality and adapter trimming (Trimgalore)
Perform FASTQC post trimming
Perform MultiQC - take both pre and post trimming FASTQC results as input.

I've tried to run the pipeline. All steps work except the final MultiQC process where I get an error:

Process `MULTIQC` input file name collision -- There are multiple input files for each of the following file names:

However, checking the output folders - there are no duplicate file names. Furthermore, I've tried to avoid any collision by naming prefixing the 1M subsampled FASTQ files with "1M-" - so they do have distinct names from the post-trimming FASTQ files.

Any help would be much appreciated!

My pipeline:

/*
 * Pipeline input parameters
 */

log.info """\
    R N A S E Q - N F   P I P E L I N E
    ===================================
    reads        : ${params.reads}
    analysisdir  : ${params.analysisdir}
    """
    .stripIndent(true)

/*
 * Sub-sample to 250k reads, 1M lines
 */
process SUBSAMPLE {
    input:
    tuple val(sample_id), path(reads)
    path parentFolder

    output:
    tuple val(sample_id), path("${parentFolder}/1M_Subsample/1M-${sample_id}_R1.fastq"), path("${parentFolder}/1M_Subsample/1M-${sample_id}_R2.fastq"), emit: reads

    script:
    """
    if [ ! -e ${parentFolder}/1M_Subsample ]; then mkdir -p ${parentFolder}/1M_Subsample; fi
    gzip -cd ${reads[0]} | head -4000000 > ${parentFolder}/1M_Subsample/1M-${sample_id}_R1.fastq
    gzip -cd ${reads[1]} | head -4000000 > ${parentFolder}/1M_Subsample/1M-${sample_id}_R2.fastq
    """
}

/*
 * Perform FastQC
 */
process FASTQC {
    input:
    tuple val(sample_id), path(read1), path(read2)
    path parentFolder

    output:
    file "${parentFolder}/1M_fastqc/*_fastqc.{zip,html}"
    
    script:
    """
    # Load module
    module load fastqc
    
    # Create output folders
    if [ ! -e ${parentFolder}/1M_fastqc ]; then mkdir -p ${parentFolder}/1M_fastqc; fi

    # FastQC files
    fastqc -o ${parentFolder}/1M_fastqc ${read1} ${read2}
    """
}

/*
 * Perform trimming
 */
process TRIMMING {
    input:
    tuple val(sample_id), path(reads)
    path parentFolder

    output:
    tuple val(sample_id), path("${parentFolder}/trimgalore_outs/*val*.fq.gz"), emit: reads
	path("${parentFolder}/trimgalore_outs/*report.txt"), optional: true, emit: report

    script:
    """
    # Load module
    module load trimgalore/0.6.5

    # Create output folders
    if [ ! -e ${parentFolder}/trimgalore_outs ]; then mkdir -p ${parentFolder}/trimgalore_outs; fi
    
	# Run trimgalore
	# Options --length, -e, --stringency, --quality are set to default
	trim_galore --quality 20 \
	--length 20 \
	-j ${NSLOTS} \
	--paired \
	--stringency 1 \
	-e 0.1 \
	--output_dir ${parentFolder}/trimgalore_outs \
	${reads}
    """
}

/*
 * Perform FastQC, post trimming
 */
process FASTQC_PT {
    input:
    tuple val(sample_id), path(reads)
    path parentFolder

    output:
    file "${parentFolder}/post_trim_fastqc/*_fastqc.{zip,html}"

    script:
    """
    # Load module
    module load fastqc

    # Create output folders
    if [ ! -e ${parentFolder}/post_trim_fastqc ]; then mkdir -p ${parentFolder}/post_trim_fastqc; fi
    
    # FastQC files
    fastqc -o ${parentFolder}/post_trim_fastqc ${reads}
    """
}

/*
 * Perform MultiQC
 */
process MULTIQC {
    conda 'multiqc=1.21'

    input:
    file '*'

    output:
    path "$params.analysisdir/multiqc_report.html"

    script:
    """
    # Run multiqc
    multiqc .
    """
}

/*
 * Define workflow
 */
workflow {
    Channel
        .fromFilePairs(params.reads, checkIfExists: true)
        .set { read_pairs_ch }
    SUBSAMPLE(read_pairs_ch, params.analysisdir)
    fastqc_ch = FASTQC(SUBSAMPLE.out.reads, params.analysisdir)
    TRIMMING(read_pairs_ch, params.analysisdir)
    fastqcPT_ch = FASTQC_PT(TRIMMING.out.reads, params.analysisdir)
    MULTIQC(fastqcPT_ch.mix(fastqc_ch).collect())
}

Answered by jamesboot

May 25, 2024

Hi, thanks for your suggestions, in the end I changed the FASTQC processes to output a file instead of a path using:

output:
file "*fastqc*"

Instead of:

output:
file "${parentFolder}/1M_fastqc/*_fastqc.{zip,html}"

Then in the workflow I defined the input for the MULTIQC process as the collection of the two channels coming out of the FASTQC processes:

MULTIQC(ch_fastqc.collect(), ch_fastqc_trim.collect())

Finally I defined the two inputs separately in the MULTIQC process:

input:
file("*")
file("*")

This has fixed the problems. Thanks for the suggestions above.

View full answer

bentsherman · 2024-05-06T14:51:49Z

bentsherman
May 6, 2024
Maintainer

I would look at how the nf-core pipelines do it. For example it looks like nf-core/rnaseq defines separate inputs for the MULTIQC process: https://github.com/nf-core/rnaseq/blob/master/modules/local/multiqc/main.nf

0 replies

jamesboot · 2024-05-21T15:19:45Z

jamesboot
May 21, 2024
Author

Thanks for the suggestions. I tried defining separate inputs for the MULTIQC process as suggested the two inputs were the two directories where the two FASTQC process outputs were saved - however I'm still getting an error:

ERROR ~ Error executing process > 'MULTIQC'

Caused by:
  Process `MULTIQC` input file name collision -- There are multiple input files for each of the following file names: 1M_fastqc, post_trim_fastqc

As before, the files in each output folder are unique and the two directory names are unique.

I also tried going back to the mix and collect channels approach and get the same error. My current current script is:

/*
 * Pipeline input parameters
 */

log.info """\
    R N A S E Q - N F   P I P E L I N E
    ===================================
    reads        : ${params.reads}
    analysisdir  : ${params.analysisdir}
    """
    .stripIndent(true)

/*
 * Sub-sample to 250k reads, 1M lines
 */
process SUBSAMPLE {
    input:
    tuple val(sample_id), path(reads)
    path parentFolder

    output:
    tuple val(sample_id), path("${parentFolder}/1M_Subsample/1M-${sample_id}_R1.fastq"), path("${parentFolder}/1M_Subsample/1M-${sample_id}_R2.fastq"), emit: reads

    script:
    """
    if [ ! -e ${parentFolder}/1M_Subsample ]; then mkdir -p ${parentFolder}/1M_Subsample; fi
    gzip -cd ${reads[0]} | head -4000000 > ${parentFolder}/1M_Subsample/1M-${sample_id}_R1.fastq
    gzip -cd ${reads[1]} | head -4000000 > ${parentFolder}/1M_Subsample/1M-${sample_id}_R2.fastq
    """
}

/*
 * Perform FastQC
 */
process FASTQC {
    input:
    tuple val(sample_id), path(read1), path(read2)
    path parentFolder

    output:
    path("${parentFolder}/1M_fastqc")
    
    script:
    """
    # Load module
    module load fastqc
    
    # Create output folders
    if [ ! -e ${parentFolder}/1M_fastqc ]; then mkdir -p ${parentFolder}/1M_fastqc; fi

    # FastQC files
    fastqc -o ${parentFolder}/1M_fastqc ${read1} ${read2}
    """
}

/*
 * Perform trimming
 */
process TRIMMING {
    input:
    tuple val(sample_id), path(reads)
    path parentFolder

    output:
    tuple val(sample_id), path("${parentFolder}/trimgalore_outs/*val*.fq.gz"), emit: reads
	path("${parentFolder}/trimgalore_outs/*report.txt"), optional: true, emit: report

    script:
    """
    # Load module
    module load trimgalore/0.6.5

    # Create output folders
    if [ ! -e ${parentFolder}/trimgalore_outs ]; then mkdir -p ${parentFolder}/trimgalore_outs; fi
    
	# Run trimgalore
	# Options --length, -e, --stringency, --quality are set to default
	trim_galore --quality 20 \
	--length 20 \
	-j ${NSLOTS} \
	--paired \
	--stringency 1 \
	-e 0.1 \
	--output_dir ${parentFolder}/trimgalore_outs \
	${reads}
    """
}

/*
 * Perform FastQC, post trimming
 */
process FASTQC_PT {
    input:
    tuple val(sample_id), path(reads)
    path parentFolder

    output:
    path("${parentFolder}/post_trim_fastqc")

    script:
    """
    # Load module
    module load fastqc

    # Create output folders
    if [ ! -e ${parentFolder}/post_trim_fastqc ]; then mkdir -p ${parentFolder}/post_trim_fastqc; fi

    # FastQC files
    fastqc -o ${parentFolder}/post_trim_fastqc ${reads}
    """
}

/*
 * Perform MultiQC
 */
process MULTIQC {
    conda 'multiqc=1.21'

    input:
    path('*')

    output:
    path("$params.analysisdir/multiqc_report.html")

    script:
    """
    # Run multiqc
    multiqc .
    """
}

/*
 * Define workflow
 */
workflow {
    Channel
        .fromFilePairs(params.reads, checkIfExists: true)
        .set { read_pairs_ch }
    SUBSAMPLE(read_pairs_ch, params.analysisdir)
    fastqc_ch = FASTQC(SUBSAMPLE.out.reads, params.analysisdir)
    TRIMMING(read_pairs_ch, params.analysisdir)
    fastqcPT_ch = FASTQC_PT(TRIMMING.out.reads, params.analysisdir)
    MULTIQC(fastqcPT_ch.mix(fastqc_ch).collect())
}

Any help would be much appreciated.

0 replies

complexgenome · 2024-05-21T21:00:17Z

complexgenome
May 21, 2024

check response by muffato, Matthieu Muffato
https://community.seqera.io/t/input-file-name-collision-there-are-multiple-input-files/660/5

0 replies

jamesboot · 2024-05-22T16:11:03Z

jamesboot
May 22, 2024
Author

Thanks, but I don't understand why this should be necessary - the outputs are in two different sub directories and are all named uniquely. This is the file list:

/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-1_S25_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-1_S25_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-1_S25_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-1_S25_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-2_S26_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-2_S26_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-2_S26_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-2_S26_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-3_S27_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-3_S27_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-3_S27_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-BLGCSEQL1084-Cntl-3_S27_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-1h_S1_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-1h_S1_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-1h_S1_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-1h_S1_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-24h_S9_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-24h_S9_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-24h_S9_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-24h_S9_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-d7_S17_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-d7_S17_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-d7_S17_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Act-d7_S17_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-1h_S5_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-1h_S5_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-1h_S5_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-1h_S5_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-24h_S13_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-24h_S13_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-24h_S13_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-24h_S13_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-d7_S21_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-d7_S21_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-d7_S21_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-1-Ind-d7_S21_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-1h_S2_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-1h_S2_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-1h_S2_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-1h_S2_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-24h_S10_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-24h_S10_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-24h_S10_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-24h_S10_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-d7_S18_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-d7_S18_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-d7_S18_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Act-d7_S18_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-1h_S6_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-1h_S6_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-1h_S6_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-1h_S6_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-24h_S14_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-24h_S14_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-24h_S14_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-24h_S14_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-d7_S22_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-d7_S22_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-d7_S22_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-2-Ind-d7_S22_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-1h_S3_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-1h_S3_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-1h_S3_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-1h_S3_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-24h_S11_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-24h_S11_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-24h_S11_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-24h_S11_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-d7_S19_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-d7_S19_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-d7_S19_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Act-d7_S19_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-1h_S7_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-1h_S7_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-1h_S7_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-1h_S7_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-24h_S15_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-24h_S15_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-24h_S15_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-24h_S15_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-d7_S23_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-d7_S23_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-d7_S23_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-3-Ind-d7_S23_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-1h_S4_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-1h_S4_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-1h_S4_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-1h_S4_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-24h_S12_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-24h_S12_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-24h_S12_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-24h_S12_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-d7_S20_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-d7_S20_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-d7_S20_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Act-d7_S20_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-1h_S8_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-1h_S8_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-1h_S8_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-1h_S8_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-24h_S16_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-24h_S16_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-24h_S16_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-24h_S16_R2_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-d7_S24_R1_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-d7_S24_R1_fastqc.zip
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-d7_S24_R2_fastqc.html
/1M_fastqc/1M-GC-JK-9047-Osc-4-Ind-d7_S24_R2_fastqc.zip
/1M_fastqc/1M-Undetermined_S0_R1_fastqc.html
/1M_fastqc/1M-Undetermined_S0_R1_fastqc.zip
/1M_fastqc/1M-Undetermined_S0_R2_fastqc.html
/1M_fastqc/1M-Undetermined_S0_R2_fastqc.zip

/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-1_S25_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-1_S25_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-1_S25_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-1_S25_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-2_S26_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-2_S26_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-2_S26_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-2_S26_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-3_S27_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-3_S27_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-3_S27_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-BLGCSEQL1084-Cntl-3_S27_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-1h_S1_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-1h_S1_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-1h_S1_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-1h_S1_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-24h_S9_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-24h_S9_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-24h_S9_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-24h_S9_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-d7_S17_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-d7_S17_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-d7_S17_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Act-d7_S17_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-1h_S5_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-1h_S5_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-1h_S5_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-1h_S5_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-24h_S13_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-24h_S13_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-24h_S13_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-24h_S13_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-d7_S21_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-d7_S21_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-d7_S21_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-1-Ind-d7_S21_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-1h_S2_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-1h_S2_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-1h_S2_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-1h_S2_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-24h_S10_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-24h_S10_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-24h_S10_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-24h_S10_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-d7_S18_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-d7_S18_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-d7_S18_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Act-d7_S18_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-1h_S6_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-1h_S6_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-1h_S6_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-1h_S6_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-24h_S14_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-24h_S14_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-24h_S14_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-24h_S14_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-d7_S22_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-d7_S22_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-d7_S22_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-2-Ind-d7_S22_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-1h_S3_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-1h_S3_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-1h_S3_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-1h_S3_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-24h_S11_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-24h_S11_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-24h_S11_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-24h_S11_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-d7_S19_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-d7_S19_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-d7_S19_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Act-d7_S19_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-1h_S7_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-1h_S7_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-1h_S7_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-1h_S7_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-24h_S15_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-24h_S15_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-24h_S15_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-24h_S15_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-d7_S23_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-d7_S23_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-d7_S23_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-3-Ind-d7_S23_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-1h_S4_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-1h_S4_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-1h_S4_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-1h_S4_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-24h_S12_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-24h_S12_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-24h_S12_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-24h_S12_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-d7_S20_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-d7_S20_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-d7_S20_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Act-d7_S20_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-1h_S8_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-1h_S8_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-1h_S8_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-1h_S8_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-24h_S16_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-24h_S16_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-24h_S16_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-24h_S16_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-d7_S24_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-d7_S24_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-d7_S24_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/GC-JK-9047-Osc-4-Ind-d7_S24_L001_R2_001_val_2_fastqc.zip
/post_trim_fastqc/Undetermined_S0_L001_R1_001_val_1_fastqc.html
/post_trim_fastqc/Undetermined_S0_L001_R1_001_val_1_fastqc.zip
/post_trim_fastqc/Undetermined_S0_L001_R2_001_val_2_fastqc.html
/post_trim_fastqc/Undetermined_S0_L001_R2_001_val_2_fastqc.zip

0 replies

complexgenome · 2024-05-22T16:14:58Z

complexgenome
May 22, 2024

@jamesboot Sorry, I do not know.

0 replies

bentsherman · 2024-05-23T12:50:17Z

bentsherman
May 23, 2024
Maintainer

I think when you have a single path('*') input, it doesn't matter if incoming files are in different directories, they will be flattened when they are linked into the task directory. So you must either specify a different path input for each subdirectory or flatten them yourself to prevent the name collision.

Improving this case is something I can look into as part of #4553

0 replies

jamesboot · 2024-05-25T15:12:01Z

jamesboot
May 25, 2024
Author

Hi, thanks for your suggestions, in the end I changed the FASTQC processes to output a file instead of a path using:

output:
file "*fastqc*"

Instead of:

output:
file "${parentFolder}/1M_fastqc/*_fastqc.{zip,html}"

Then in the workflow I defined the input for the MULTIQC process as the collection of the two channels coming out of the FASTQC processes:

MULTIQC(ch_fastqc.collect(), ch_fastqc_trim.collect())

Finally I defined the two inputs separately in the MULTIQC process:

input:
file("*")
file("*")

This has fixed the problems. Thanks for the suggestions above.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MultiQC: Process `MULTIQC` input file name collision #4969

{{title}}

Replies: 7 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

MultiQC: Process MULTIQC input file name collision #4969

jamesboot May 3, 2024

Replies: 7 comments

bentsherman May 6, 2024 Maintainer

jamesboot May 21, 2024 Author

complexgenome May 21, 2024

jamesboot May 22, 2024 Author

complexgenome May 22, 2024

bentsherman May 23, 2024 Maintainer

jamesboot May 25, 2024 Author

MultiQC: Process `MULTIQC` input file name collision #4969

jamesboot
May 3, 2024

bentsherman
May 6, 2024
Maintainer

jamesboot
May 21, 2024
Author

complexgenome
May 21, 2024

jamesboot
May 22, 2024
Author

complexgenome
May 22, 2024

bentsherman
May 23, 2024
Maintainer

jamesboot
May 25, 2024
Author