Skip to content

Fix/prevent falco to process empty fastq files#5731

Closed
glichtenstein wants to merge 44 commits intonf-core:5612-bcl-demultiplexfrom
glichtenstein:fix/prevent-falco-to-process-empty-fastq-files
Closed

Fix/prevent falco to process empty fastq files#5731
glichtenstein wants to merge 44 commits intonf-core:5612-bcl-demultiplexfrom
glichtenstein:fix/prevent-falco-to-process-empty-fastq-files

Conversation

@glichtenstein
Copy link
Copy Markdown
Contributor

PR checklist

What about doing the filter directly on the falco module main.nf. This way the subworkflow/bcl_demultiplex/main.nf does not have to handle this issue nor the workflow/demultiplex.nf
I have used a file size filter, to avoid inserting in the falco list of ${reads} those files that have less than 20bytes.
Rationale: An empty text file compressed in tar.gz is usually 20 bytes in size.

Closes #5638

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the module conventions in the contribution docs
  • If necessary, include test data in your PR.
  • Remove all TODO statements.
  • Emit the versions.yml file.
  • Follow the naming conventions.
  • Follow the parameters requirements.
  • Follow the input/output options guidelines.
  • Add a resource label
  • Use BioConda and BioContainers if possible to fulfil software requirements.
  • Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
    • For modules:
      • nf-core modules test <MODULE> --profile docker
      • nf-core modules test <MODULE> --profile singularity
      • nf-core modules test <MODULE> --profile conda
    • For subworkflows:
      • nf-core subworkflows test <SUBWORKFLOW> --profile docker
      • nf-core subworkflows test <SUBWORKFLOW> --profile singularity
      • nf-core subworkflows test <SUBWORKFLOW> --profile conda

famosab and others added 30 commits May 16, 2024 08:25
* port to nf-test

* update tests to include versions
* Add missing tximport outputs

* Update snapshot for tximeta/tximport
Replace multiple ifs with a single for

The loop checks if files exist before moving them to temp/
* port to nf-test

* update snap

* update tests to include versions

* port to nf-test

* update snap

* update tests to include versions

* change snapshot to read the dict

* check file md5sum

* Apply suggestions from code review

* update snap

---------

Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io>
* port to nf-test

* include versions in test

* port to nf-test

* include versions in test

* change to check tables

* change tests to assert whole output and update snap

* generalize assertions
* add index

* add bwameme_mem

* Update modules/nf-core/bwameme/index/tests/main.nf.test

Co-authored-by: Annick Renevey <47788523+rannick@users.noreply.github.com>

---------

Co-authored-by: Annick Renevey <47788523+rannick@users.noreply.github.com>
* use conda-forge instead of anaconda

* Update modules/nf-core/homer/maketagdirectory/environment.yml

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* nf-test homer/findpeaks

* nf-test homer/maketagdirectory

* nf-test homer/pos2bed

* fix perl version?

* Update modules/nf-core/gawk/environment.yml

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* fix gawk

---------

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>
The new container version no longer requires overriding the entrypoint.
* Added stub for sratools

* Added command echo for fasterqdump
* Version bumped bwa/index and bwa/aln

* Removed bam md5sum from snapshot
* Bump bwa mem version

* Most recent htslib
* add mmseqs_contig_taxonomy

* apply prettier

* code linting

* changed the test dataset to bacteroides_fragilis instead of influenzae

* give more memory to mmseqs-taxa

* memory increased to 16.GB

* increase the process mem requirements

* fix memory

* add reviewer suggestion

* change back memory

* fix snap

* fix snap
…-core#5598)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…f-core#5636)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
)

* Add stub run and bump version 0.30.0

* Add nf-test

* Delete pytest

* Remove tag from pytest_modules
* add rrnatransctipts

* add util for extracting ribosomal transcripts

* fix tests

* fix tests

* mixup with ucsc gtftogenepred

* mixup with ucsc gtftogenepred

* mixup with ucsc gtftogenepred

* make output optional

* fill meta

* reset gtftogenepred

* use grep instead of template

* fix

* fix test so if grep does not find anything, the exit code is still 0

---------

Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
…ow (nf-core#5628)

* Return primary bam and index file in fastq_align_bamcmp_bwa subworkflow and update snapshots.

* Add new tests and update snapshots.

* Remove opt out for conda for fastq_align_bamcmp_bwa

* Update snapshots

* Restore opt out for conda for fastq_align_bamcmp_bwa

---------

Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
* Author: @MartaSantanaSilva
Initial pull for module grabix/check
Addition of one test (total: 1)

* Update modules/nf-core/grabix/check/main.nf

Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com>

* Update modules/nf-core/grabix/check/main.nf

Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com>

* Update modules/nf-core/grabix/check/main.nf

Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com>

* Update modules/nf-core/grabix/check/meta.yml

Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com>

* new module: grabix
Fixes nf-core#5181

* Apply suggestions from code review

* Update modules/nf-core/grabix/check/meta.yml

* fix conda declaration

* update snapshot

* prettier

---------

Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com>
Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io>
Co-authored-by: maxulysse <max.u.garcia@gmail.com>
Parabricks doesn't require stageInMode to be 'copy'

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>
* chore(deps): update biocontainers/glimpse-bio docker tag to v2.0.1

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* The channel was renamed in nf-core#5117

* The Conda yml files need to match the container definitions

* Typo ! There was a missing s

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Louis LE NEZET <58640615+LouisLeNezet@users.noreply.github.com>
Co-authored-by: Matthieu Muffato <mm49@sanger.ac.uk>
…core#5653)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* Bump csvtk new version

* Add nf-test

* Get rid of pytest

* Bump version in singularity images

* Update tests/config/pytest_modules.yml

---------

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
* Stub and nf-test for fastk/fastk

* Added chmod +r to add read permissions for docker

* Added a+r
* Remove leftover of previous PR

* Bump version 0.30.0 of csvtk

* Ad nf-test for csvtk/split

* Remove csvtk/split from pytest_modules.yml

* Make lint happy

* Update path on tests

* Update modules/nf-core/csvtk/split/tests/main.nf.test

* Update modules/nf-core/csvtk/split/tests/main.nf.test
…f-core#5662)

* Remove cp operation from bclconvert

Replaces cp operation with native Nextflow output path handling.

Fixes nf-core#5644

* Update modules/nf-core/bclconvert/main.nf
* working on the main.nf and meta.yml initially

* working on the main.nf and meta.yml initially

* Adding the updates to meta.yml

* Adding the latest code changes that add a working version of the tests

* Fixing the version mismatch in the docker and singularity containers

* Removing the trailing whitespaces

* Bump parabricks 4.3.0-1 + remove stageInMode (nf-core#5646)

Parabricks doesn't require stageInMode to be 'copy'

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* glimpse2 version bump - abandoned (nf-core#4236)

* chore(deps): update biocontainers/glimpse-bio docker tag to v2.0.1

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* The channel was renamed in nf-core#5117

* The Conda yml files need to match the container definitions

* Typo ! There was a missing s

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Louis LE NEZET <58640615+LouisLeNezet@users.noreply.github.com>
Co-authored-by: Matthieu Muffato <mm49@sanger.ac.uk>

* Update pre-commit hook renovatebot/pre-commit-hooks to v37.371.1 (nf-core#5653)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* Bump csvtk version to CSVTK/JOIN  and add nf-test (nf-core#5642)

* Bump csvtk new version

* Add nf-test

* Get rid of pytest

* Bump version in singularity images

* Update tests/config/pytest_modules.yml

---------

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

* Added task.memory to meryl modules and .meryl ext for outputs (nf-core#5655)

* Stub and nf-test for fastk/fastk (nf-core#5631)

* Stub and nf-test for fastk/fastk

* Added chmod +r to add read permissions for docker

* Added a+r

* Bump csvtk version to CSVTK/SPLIT and add nf-test (nf-core#5656)

* Remove leftover of previous PR

* Bump version 0.30.0 of csvtk

* Ad nf-test for csvtk/split

* Remove csvtk/split from pytest_modules.yml

* Make lint happy

* Update path on tests

* Update modules/nf-core/csvtk/split/tests/main.nf.test

* Update modules/nf-core/csvtk/split/tests/main.nf.test

* Update modules/nf-core/viennarna/rnafold/tests/main.nf.test

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

* Adding the corrected version of meta.yml file

---------

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>
Co-authored-by: Marcel Ribeiro-Dantas <marcel.ribeirodantas@seqera.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Louis LE NEZET <58640615+LouisLeNezet@users.noreply.github.com>
Co-authored-by: Matthieu Muffato <mm49@sanger.ac.uk>
Co-authored-by: Jose Espinosa-Carrasco <kadomu@gmail.com>
Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
Co-authored-by: Usman Rashid <usman@smme.edu.pk>
* Add nf-test to chromap/index

* Updated snapshot and removed old tests

* Fix chromap versions and add tags

* [automated] Fix linting with Prettier

* Check genome.index exists instead of match

Co-authored-by: Jose Espinosa-Carrasco <kadomu@gmail.com>

* snapshot fix

* Updated snapshot

* Add version to snapshot

* Change version to versions in snapshot

---------

Co-authored-by: nf-core-bot <core@nf-co.re>
Co-authored-by: Jose Espinosa-Carrasco <kadomu@gmail.com>
KamilMaliszArdigen and others added 14 commits May 22, 2024 19:24
* Added nf-test for gatk4/indexfeaturefile

* Deleted pytest gatk4/indexfeaturefile tests
* Add tests for Salsa

* Update test file

* Delete unused test file

* Resolve linting error

* Remove unused params section

* Update modules/nf-core/salsa2/tests/main.nf.test

Co-authored-by: Sateesh_Peri <33637490+sateeshperi@users.noreply.github.com>

* Use snapshot and match methods

* Update snapeshot, compare files

* Remove pytests

---------

Co-authored-by: James Kaptuch <jkaptuch@ambrygen.com>
Co-authored-by: Sateesh_Peri <33637490+sateeshperi@users.noreply.github.com>
* biscuit/align nf-test and snapshot

* Update modules/nf-core/biscuit/align/tests/main.nf.test

* remove bam from snapshots

* remove bam,bai from snapshots

---------

Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
* Adding the partially working version of RNAcofold code

* Adding the rnacofold tests and main

* Removing comments from the env.yml file

* Update modules/nf-core/viennarna/rnacofold/main.nf

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

* Update modules/nf-core/viennarna/rnacofold/tests/main.nf.test

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

* Adding the meta to track sample information

---------

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
* Added nf-test for gatk4/intervallisttools

* Update modules/nf-core/gatk4/intervallisttools/tests/main.nf.test

Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>

* Added nf-test stub for gatk4/intervallisttools

* Updated snap nf-test stub for gatk4/intervallisttools

---------

Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
* working on the main.nf and meta.yml initially

* working on the main.nf and meta.yml initially

* Adding the updates to meta.yml

* Adding the latest code changes that add a working version of the tests

* Fixing the version mismatch in the docker and singularity containers

* Removing the trailing whitespaces

* Bump parabricks 4.3.0-1 + remove stageInMode (nf-core#5646)

Parabricks doesn't require stageInMode to be 'copy'

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* glimpse2 version bump - abandoned (nf-core#4236)

* chore(deps): update biocontainers/glimpse-bio docker tag to v2.0.1

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* Update main.nf

* The channel was renamed in nf-core#5117

* The Conda yml files need to match the container definitions

* Typo ! There was a missing s

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Louis LE NEZET <58640615+LouisLeNezet@users.noreply.github.com>
Co-authored-by: Matthieu Muffato <mm49@sanger.ac.uk>

* Update pre-commit hook renovatebot/pre-commit-hooks to v37.371.1 (nf-core#5653)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* Bump csvtk version to CSVTK/JOIN  and add nf-test (nf-core#5642)

* Bump csvtk new version

* Add nf-test

* Get rid of pytest

* Bump version in singularity images

* Update tests/config/pytest_modules.yml

---------

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

* Added task.memory to meryl modules and .meryl ext for outputs (nf-core#5655)

* Stub and nf-test for fastk/fastk (nf-core#5631)

* Stub and nf-test for fastk/fastk

* Added chmod +r to add read permissions for docker

* Added a+r

* Bump csvtk version to CSVTK/SPLIT and add nf-test (nf-core#5656)

* Remove leftover of previous PR

* Bump version 0.30.0 of csvtk

* Ad nf-test for csvtk/split

* Remove csvtk/split from pytest_modules.yml

* Make lint happy

* Update path on tests

* Update modules/nf-core/csvtk/split/tests/main.nf.test

* Update modules/nf-core/csvtk/split/tests/main.nf.test

* Update modules/nf-core/viennarna/rnafold/tests/main.nf.test

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

* Adding the working version of ViennaRNA/RNALfold module

* Adding the corrected version of meta.yml file

* Removing trailing whitespaces in all files

* Removing the type with backtick

* Removing the whitespace from rnafold main.nf

* Removing the viennarna/rnafold and removing the comments and --- from env.yml

* Update modules/nf-core/viennarna/rnalfold/main.nf

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>

* Update modules/nf-core/viennarna/rnalfold/main.nf

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>

* Update modules/nf-core/viennarna/rnalfold/main.nf

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>

* Update modules/nf-core/viennarna/rnalfold/tests/main.nf.test

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>

* Update modules/nf-core/viennarna/rnalfold/tests/main.nf.test

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>

* Update modules/nf-core/viennarna/rnalfold/tests/nf-test.config

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>

* Deleting nf-test.config for viennarna/rnalfold as per suggested

* Adding the correct test assertions and snapshots

---------

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>
Co-authored-by: Marcel Ribeiro-Dantas <marcel.ribeirodantas@seqera.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Louis LE NEZET <58640615+LouisLeNezet@users.noreply.github.com>
Co-authored-by: Matthieu Muffato <mm49@sanger.ac.uk>
Co-authored-by: Jose Espinosa-Carrasco <kadomu@gmail.com>
Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
Co-authored-by: Usman Rashid <usman@smme.edu.pk>
Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
* Add checkm/lineagewf nf tests.

* Remove files contain timestamp from snapshot

* Remove pytest files

* Delete checkm_lineagewf entry in pytest_modules.yml

* Apply suggestions from code review

Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>

---------

Co-authored-by: Wei-An Chen <wchen@VEDANTABIO.COM>
Co-authored-by: Sateesh_Peri <33637490+sateeshperi@users.noreply.github.com>
Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io>
Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>
* added module 10xbamtofastq

* remove extra test file

* Update main.nf.test

* Update main.nf

* Rename module and fix tests

---------

Co-authored-by: Harshil Patel <drpatelhh@gmail.com>
Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
* Added nf-test stub for gatk4/variantfiltration

* Update for lint

* Update for lint
* Add nf-test to chromap/chromap

* Add nf-test to chromap/chromap (tags added)

* Update modules/nf-core/chromap/chromap/tests/main.nf.test

* Update modules/nf-core/chromap/chromap/tests/tags.yml

* Update pytest_modules.yml

* Update main.nf.test

* Update main.nf.test.snap

* Update main.nf.test

* Update main.nf.test.snap

---------

Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io>
Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>
… for merqury/hapmers (nf-core#5657)

* Migrated to nf-test

* Moved to merqury/merqury

* Fixed test issues

* Moved pngs to name check

* Added version warn and trio mode test
* Swap ngscheckmate/ncm to nftest

* Update main.nf.test
* Initial work towards a module for RTN TNI

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Make it a simple PoC

Tests are passing. From now on, I can make it more complex and
closer to what I want this module to do in the end.

* Add two more steps in the module

* Prettify file

* Use test data, add remaining steps/collect output in TNI

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Fix DOI in meta.yml

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Some ongoing changes

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Rename rtn_tni.R to rtn_tni.r

* Fix snapshot mismatch issue

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Add --n_permutations + tests

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Get rid of the TODOs

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Use different dataset (real gene ids)

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Add support to --tfs + tests

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Fix indentation for ECLint check

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

* Remove quotes from environment dependencies for conda

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>

---------

Signed-off-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io>
@glichtenstein glichtenstein deleted the fix/prevent-falco-to-process-empty-fastq-files branch May 31, 2024 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: No status

Development

Successfully merging this pull request may close these issues.