Add tests, use ena-webin-cli handler, refactor modules.conf and update docs (#39)

Features:
* replace `ena-webin-cli` with `ena-webin-cli handler` 
* added `webin-cli.jar` download to reuse it for every upload
* webin-cli-wrapper outputs TSV with accessions which resolves #31 
* FASTAVALIDATOR step added to genomesubmit (previously we only validated fasta in assemblysubmit)
Tests:
* multiple tests added along with snapshots and profiles to test` --mode mag` and `--mode metagenomic_assembly`, test data pushed to nf-datasets
* more info about tests #32 (comment)
Bug fixes: 
* added `triggers` to only download DBs if there are data to process and no local DB provided, 
* create output folder for metadata CSV/TSV if it doesn't exist, 
* resolved OUT_OF_MEM problem in webin-cli-wrapper,
* path in metadata CSV/TSV leading to http locations for remote files
* solves missing secrets issue #45 
* solves problem with nf-tests on GitHub actions #45 
Other:
* massive docs update
* refactor `modules.config` and clean up published results
* WEBIN_ACCOUNT renamed to ENA_WEBIN, WEBIN_PASSWORD to ENA_WEBIN_PASSWORD

Author: ochkalova

.github/actions/nf-test/action.yml

@@ -56,6 +56,12 @@ runs:
         channel-priority: strict
         conda-remove-defaults: true
 
+    - name: Configure Nextflow secrets
+      shell: bash
+      run: |
+        nextflow secrets set ENA_WEBIN "$WEBIN_ACCOUNT"
+        nextflow secrets set ENA_WEBIN_PASSWORD "$WEBIN_PASSWORD"
+
     - name: Run nf-test
       shell: bash
       env:

CITATIONS.md

@@ -14,6 +14,40 @@
 
 > Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.
 
+- [CoverM](https://github.com/wwood/CoverM)
+
+> Aroney ST, Newell RJ, Nissen JN, Camargo AP, Tyson GW, Woodcroft BJ. CoverM: Read alignment statistics for metagenomics. Bioinformatics. 2025;41(4):btaf147. doi: 10.1093/bioinformatics/btaf147. PubMed PMID: 40193404; PubMed Central PMCID: PMC11993303.
+
+- [CheckM2](https://github.com/chklovski/CheckM2)
+
+> Chklovski A, Parks DH, Woodcroft BJ, Tyson GW. CheckM2: a rapid, scalable and accurate tool for assessing microbial genome quality using machine learning. Nat Methods. 2023;20(8):1203-1212. doi: 10.1038/s41592-023-01940-w. PubMed PMID: 37500759; PubMed Central PMCID: not available.
+
+- [CAT and BAT](https://doi.org/10.1186/s13059-019-1817-x)
+
+> von Meijenfeldt FAB, Arkhipova K, Cambuy DD, Coutinho FH, Dutilh BE. Robust taxonomic classification of uncharted microbial sequences and bins with CAT and BAT. Genome Biol. 2019;20(1):217. doi: 10.1186/s13059-019-1817-x. PubMed PMID: 31640809; PubMed Central PMCID: PMC6805573.
+
+- [tRNAscan-SE 2.0](https://doi.org/10.1093/nar/gkab688)
+
+> Chan PP, Lin BY, Mak AJ, Lowe TM. tRNAscan-SE 2.0: Improved detection and functional classification of transfer RNA genes. Nucleic Acids Res. 2021;49(16):9077-9096. doi: 10.1093/nar/gkab688. PubMed PMID: 34417604; PubMed Central PMCID: PMC8450103.
+
+- [barrnap](https://github.com/tseemann/barrnap)
+
+  > Seemann T. Barrnap: rapid ribosomal RNA prediction. GitHub repository. https://github.com/tseemann/barrnap
+
+## Submission and helper tools
+
+- [ENA Webin-CLI](https://github.com/enasequence/webin-cli)
+
+  > European Nucleotide Archive. Webin command line submission interface (Webin-CLI). GitHub repository. https://github.com/enasequence/webin-cli
+
+- [assembly_uploader](https://github.com/EBI-Metagenomics/assembly_uploader)
+
+  > EBI Metagenomics. ENA Metagenome Assembly uploader. GitHub repository. https://github.com/EBI-Metagenomics/assembly_uploader
+
+- [genome_uploader](https://github.com/EBI-Metagenomics/genome_uploader)
+
+  > EBI Metagenomics. ENA public Bins and MAGs uploader. GitHub repository. https://github.com/EBI-Metagenomics/genome_uploader
+
 ## Software packaging/containerisation tools
 
 - [Anaconda](https://anaconda.com)

README.md

@@ -38,9 +38,9 @@ Currently, the pipeline supports three submission modes, each routed to a dedica
 
 Setup your environment secrets before running the pipeline:
 
-`nextflow secrets set WEBIN_ACCOUNT "Webin-XXX"`
+`nextflow secrets set ENA_WEBIN "Webin-XXX"`
 
-`nextflow secrets set WEBIN_PASSWORD "XXX"`
+`nextflow secrets set ENA_WEBIN_PASSWORD "XXX"`
 
 Make sure you update commands above with your authorised credentials.
 
@@ -55,43 +55,52 @@ The input must follow `assets/schema_input_genome.json`.
 Required columns:
 
 - `sample`
-- `fasta` (must end with `.fa.gz` or `.fasta.gz`)
+- `fasta` (must end with `.fa.gz`, `.fasta.gz`, or `.fna.gz`)
 - `accession`
 - `assembly_software`
 - `binning_software`
 - `binning_parameters`
-- `stats_generation_software`
 - `metagenome`
 - `environmental_medium`
 - `broad_environment`
 - `local_environment`
 - `co-assembly`
 
-Columns that required for now, but will be optional in the nearest future:
+At least one of the following must be provided per row:
 
+- reads (`fastq_1`, optional `fastq_2` for paired-end)
+- `genome_coverage`
+
+Additional supported columns:
+
+- `stats_generation_software`
 - `completeness`
 - `contamination`
-- `genome_coverage`
 - `RNA_presence`
 - `NCBI_lineage`
 
-Those fields are metadata required for [genome_uploader](https://github.com/EBI-Metagenomics/genome_uploader) package.
+If `genome_coverage`, `stats_generation_software`, `completeness`, `contamination`, `RNA_presence`, or `NCBI_lineage` are missing, the workflow can calculate or infer them when the required inputs are available.
+
+Those fields are metadata required for the [genome_uploader](https://github.com/EBI-Metagenomics/genome_uploader) package.
 
-Example `samplesheet_genome.csv`:
+Example `samplesheet_genomes.csv`:
 
 ```csv
-sample,fasta,accession,assembly_software,binning_software,binning_parameters,stats_generation_software,completeness,contamination,genome_coverage,metagenome,co-assembly,broad_environment,local_environment,environmental_medium,RNA_presence,NCBI_lineage
-lachnospira_eligens,data/bin_lachnospira_eligens.fa.gz,SRR24458089,spades_v3.15.5,metabat2_v2.6,default,CheckM2_v1.0.1,61.0,0.21,32.07,sediment metagenome,No,marine,cable_bacteria,marine_sediment,No,d__Bacteria;p__Proteobacteria;s_unclassified_Proteobacteria
+sample,fasta,accession,fastq_1,fastq_2,assembly_software,binning_software,binning_parameters,stats_generation_software,completeness,contamination,genome_coverage,metagenome,co-assembly,broad_environment,local_environment,environmental_medium,RNA_presence,NCBI_lineage
+lachnospira_eligens,data/bin_lachnospira_eligens.fa.gz,SRR24458089,,,spades_v3.15.5,metabat2_v2.6,default,CheckM2_v1.0.1,61.0,0.21,32.07,sediment metagenome,No,marine,cable_bacteria,marine_sediment,No,d__Bacteria;p__Proteobacteria;s__unclassified_Proteobacteria
 ```
 
+> [!IMPORTANT]
+> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown.
+
 ### `metagenomic_assemblies` mode (`ASSEMBLYSUBMIT`)
 
 The input must follow `assets/schema_input_assembly.json`.
 
 Required columns:
 
 - `sample`
-- `fasta` (must end with `.fa.gz` or `.fasta.gz`)
+- `fasta` (must end with `.fa.gz`, `.fasta.gz`, or `.fna.gz`)
 - `run_accession`
 - `assembler`
 - `assembler_version`
@@ -111,6 +120,9 @@ assembly_1,data/contigs_1.fasta.gz,data/reads_1.fastq.gz,data/reads_2.fastq.gz,,
 assembly_2,data/contigs_2.fasta.gz,,,42.7,ERR011323,MEGAHIT,1.2.9
 ```
 
+> [!IMPORTANT]
+> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown.
+
 ## Usage
 
 > [!NOTE]
@@ -122,6 +134,10 @@ All data submitted through this pipeline must be associated with an ENA study (p
 
 See the [usage documentation](docs/usage.md#submission-study) for more details.
 
+### Database setup (`CheckM2` and `CAT_pack`)
+
+The `mags`/`bins` workflow requires databases for completeness/contamination estimation and taxonomy assignment. See [Usage documentation](usage.md) for details.
+
 ### Required parameters:
 
 | Parameter                                  | Description                                                                                                       |
@@ -137,7 +153,7 @@ See the [usage documentation](docs/usage.md#submission-study) for more details.
 | Parameter           | Description                                                                              |
 | ------------------- | ---------------------------------------------------------------------------------------- |
 | `--upload_tpa`      | Flag to control the type of assembly study (third party assembly or not). Default: false |
-| `--test_upload`     | Upload to TEST ENA server instead of LIVE. Default: false                                |
+| `--test_upload`     | Upload to TEST ENA server instead of LIVE. Default: true                                 |
 | `--webincli_submit` | If set to false, submissions will be validated, but not submitted. Default: true         |
 
 General command template:
@@ -202,8 +218,8 @@ For more details and further functionality, please refer to the [usage documenta
 
 Key output locations in `--outdir`:
 
-- `upload/manifests/`: generated manifest files for submission
-- `upload/webin_cli/`: ENA Webin CLI reports
+- `mags/` or `bins/`: genome metadata, manifests, and per-sample submission support files
+- `metagenomic_assemblies/`: assembly metadata CSVs and per-sample coverage files
 - `multiqc/`: MultiQC summary report
 - `pipeline_info/`: execution reports, trace, DAG, and software versions
 

assets/samplesheet_genomes.csv

@@ -1,3 +1,3 @@
-sample,fasta,accession,fastq_1,fastq_2,assembly_software,binning_software,binning_parameters,stats_generation_software,completeness,contamination,genome_coverage,metagenome,co-assembly,broad_environment,local_environment,environmental_medium,rRNA_presence,NCBI_lineage
-lachnospira_eligens,https://github.com/nf-core/test-datasets/raw/seqsubmit/test_data/bins/bin_lachnospira_eligens.fa.gz,SRR24458089,spades_v3.15.5,mags_v1,default,CheckM2_v1.0.1,61.0,0.21,32.07,sediment metagenome,False,marine,cable bacteria,marine sediment,False,d__Bacteria;p__Proteobacteria;c__Deltaproteobacteria;o__Desulfobacterales;f__Desulfobulbaceae;g__Candidatus Electrothrix;s__
-lachnospiraceae,https://github.com/nf-core/test-datasets/raw/seqsubmit/test_data/bins/bin_lachnospiraceae.fa.gz,SRR24458087,spades_v3.15.5,mags_v1,default,CheckM2_v1.0.1,92.81,1.09,66.04,sediment metagenome,False,marine,cable bacteria,marine sediment,False,d__Bacteria;p__Proteobacteria;c__Deltaproteobacteria;o__Desulfobacterales;f__Desulfobulbaceae;g__Candidatus Electrothrix;s__Candidatus Electrothrix marina
+sample,fasta,accession,fastq_1,fastq_2,assembly_software,binning_software,binning_parameters,stats_generation_software,completeness,contamination,genome_coverage,metagenome,co-assembly,broad_environment,local_environment,environmental_medium,RNA_presence,NCBI_lineage
+lachnospira_eligens,https://github.com/nf-core/test-datasets/raw/seqsubmit/test_data/bins/bin_lachnospira_eligens.fa.gz,SRR24458089,,,spades_v3.15.5,mags_v1,default,CheckM2_v1.0.1,61.0,0.21,32.07,sediment metagenome,No,marine,cable bacteria,marine sediment,No,d__Bacteria;p__Proteobacteria;c__Deltaproteobacteria;o__Desulfobacterales;f__Desulfobulbaceae;g__Candidatus Electrothrix;s__
+lachnospiraceae,https://github.com/nf-core/test-datasets/raw/seqsubmit/test_data/bins/bin_lachnospiraceae.fa.gz,SRR24458087,,,spades_v3.15.5,mags_v1,default,CheckM2_v1.0.1,92.81,1.09,66.04,sediment metagenome,No,marine,cable bacteria,marine sediment,No,d__Bacteria;p__Proteobacteria;c__Deltaproteobacteria;o__Desulfobacterales;f__Desulfobulbaceae;g__Candidatus Electrothrix;s__Candidatus Electrothrix marina

assets/schema_input_assembly.json

@@ -17,8 +17,8 @@
                 "type": "string",
                 "format": "file-path",
                 "exists": true,
-                "pattern": "^([\\S\\s]*\\/)?[^\\s\\/]+\\.f(ast)?a\\.gz$",
-                "errorMessage": "FASTA file must be provided and have extension '.fa', '.fasta', '.fas', '.fna' (optionally gzipped)",
+                "pattern": "^([\\S\\s]*\\/)?[^\\s\\/]+\\.(fa|fasta|fna)\\.gz$",
+                "errorMessage": "FASTA file must be provided and have extension '.fa.gz', '.fasta.gz', '.fna.gz'",
                 "description": "Metagenomic assembly FASTA file"
             },
             "fastq_1": {

assets/schema_input_genome.json

@@ -17,8 +17,8 @@
                 "type": "string",
                 "format": "file-path",
                 "exists": true,
-                "pattern": "^([\\S\\s]*\\/)?[^\\s\\/]+\\.f(ast)?a\\.gz$",
-                "errorMessage": "FASTA file for sequences 1 must be provided, cannot contain spaces and must have extension '.fa.gz' or '.fasta.gz'",
+                "pattern": "^([\\S\\s]*\\/)?[^\\s\\/]+\\.(fa|fasta|fna)\\.gz$",
+                "errorMessage": "FASTA file for sequences 1 must be provided, cannot contain spaces and must have extension '.fa.gz', '.fasta.gz', or '.fna.gz'",
                 "description": "MAG/bin sequence file"
             },
             "accession": {
@@ -117,6 +117,7 @@
         "required": [
             "sample",
             "fasta",
+            "accession",
             "assembly_software",
             "co-assembly",
             "binning_software",

conf/base.config

@@ -10,7 +10,6 @@
 
 process {
 
-    // TODO nf-core: Check the defaults for all processes
     cpus   = { 1      * task.attempt }
     memory = { 6.GB   * task.attempt }
     time   = { 4.h    * task.attempt }
@@ -24,8 +23,6 @@ process {
     //        These labels are used and recognised by default in DSL2 files hosted on nf-core/modules.
     //        If possible, it would be nice to keep the same label naming convention when
     //        adding in your local modules too.
-    // TODO nf-core: Customise requirements for specific processes.
-    // See https://www.nextflow.io/docs/latest/config.html#config-process-selectors
     withLabel:process_single {
         cpus   = { 1                   }
         memory = { 6.GB * task.attempt }