eggd_TSO500

What does this app do?

Runs the Illumina TSO500 local analysis app.

What inputs are required for this app to run?

Required

• TSO500_ruo (file) - a zip file (originally provided by Illumina) containing the TSO500 local analysis app.
• run_tar_data (array:files) - raw sequencing data in tar files

Optional

• samplesheet (file) - samplesheet to use for demultiplexing and analysis, will use the one in the run data if not provided
• demultiplex_only (bool) - flag to demultiplex a runfolder and upload the fastq and log files
• upload_demultiplex_output (bool; default: true): controls if to upload the demultiplexing output files
• analysis_options (str) - a string which can be passed to the ruo command line
• isNovaSeq (bool; default: true) - passes the -isNovaSeq flag to the TSO500 local app for running on NovaSeq data
• scatter_instance (str): DNAnexus instance type to use for the per sample analysis (default: mem1_ssd1_v2_x36)
• include_samples (str) - comma separated string of sample names / regex patterns to run analyses for (mutually exclusive with exclude_samples)
• exclude_samples (str) - comma separated string of sample names / regex patterns to NOT run analyses for (mutually exclusive with include_samples)
• n_samples (int) - maximum number of samples from samplesheet to run analysis on (this will take the first n sample rows from the samplesheet)

How does this app work?

The app runs the TSO500 local app in the 'scatter / gather' mode (explained on page 9 here), this works by splitting off the per sample analysis into separate sub jobs in parallel and then combining the output in the parent job to produce the final results. This greatly speeds up analysis vs running the local app sequentially on all samples. The general outline is as follows:

• download and unpack the TSO500 resources zip file
• download and unpack the run data tar files
• parse samplesheet and (optionally) include/exclude sample rows if -iinclude_samples or -iexclude_samples specified
• run demultiplexing
• launch sub jobs per sample to run the initial analysis
• hold parent job until all scatter jobs complete
• gather up scatter job outputs and run final gather analysis step
• upload all final output files and set app output spec

What does this app output

This outputs the full TSO500 local app, including all analysis files and intermediate logs. Some of the sample analysis files are first uploaded to specific fields in the output spec for the app, these include:

Field	Type	Path	Notes
fastqs	FASTQ	/demultiplexOutput/Logs_Intermediates/FastqGeneration/	The FASTQs as output from bclconvert
dna_bams	BAM	/scatter/StitchedRealigned/*/	The 'final' BAM files upon which variant calling is performed
dna_bam_index	BAI	/scatter/StitchedRealigned/*/	Index of the above BAMs
rna_bam	BAM	/scatter/RnaAlignment/*/	Downsampled and trimmed BAMs aligned by STAR aligner
rna_bam_index	BAI	/scatter/RnaAlignment/*/	Index of the above BAMs
msi_metrics	JSON	/scatter/Msi/*/	MSI metrics in JSON format
tmb_metrics	JSON	/scatter/Tmb/*/	TMB metrics in JSON format
fusions	CSV	/gather/Results/*/	CSVs of all identified fusions from RNA analysis
small_variant_annotation	JSON	/gather/Results/*/	Annotated variants in JSON
splice_variant_annotation	JSON	/gather/Results/*/	Annotated splice variants from RNA analysis in JSON
cvo	TSV	/gather/Results/*/	CombinedVariantOutput TSV files
metricsOutput	TSV	/	Final MetricsOutput TSV file for the run
cnv_vcfs	VCF	/gather/Results/*/	Final CNV VCFs from Results directory
gvcfs	VCF	/gather/Results/*/	Genome VCFs from the merging step
splice_variants_vcf	VCF	/gather/Results/*/	VCFs of splice variants from RNA analysis

Other demultiplexing files (i.e logs etc) are uploaded and attributed to the demultiplex_logs output field, and all other analysis files are uploaded and attributed to the analysis_folder output field.

The general output directory structure is (n.b. per sample directories / files have been limited to one sample for brevity):

output_folder/
├── scatter
│   ├── Annotation
│   │   └── sample1
│   │       ├── sample1_SmallVariants_Annotated.json.gz
│   │       └── sample1_SmallVariants_Annotated.json.gz.jsi
│   ├── DnaQCMetrics
│   │   └── sample1
│   │       ├── sample1.aligned.metrics.json
│   │       ├── sample1.cnv.metrics.json
│   │       ├── sample1.collapsed.metrics.json
│   │       └── sample1.stitched.metrics.json
│   ├── Contamination
│   │   └── sample1
│   │       └── sample1.contamination.json
│   ├── StitchedRealigned
│   │   └── sample1
│   │       ├── GeminiMultiLogs
│   │       │   └── GeminiMultiOptions.used.json
│   │       ├── sample1.bam
│   │       └── sample1.bam.bai
│   ├── VariantMatching
│   │   └── sample1
│   │       └── sample1_MergedSmallVariants.genome.vcf.gz
│   ├── PhasedVariants
│   │   └── sample1
│   │       ├── PsaraLogs
│   │       │   └── PsaraOptions.used.json
│   │       ├── ScyllaLogs
│   │       │   └── ScyllaOptions.used.json
│   │       ├── sample1.Complex.vcf.gz
│   │       └── sample1_SmallVariants.filtered.genome.vcf.gz
│   ├── Msi
│   │   └── sample1
│   │       ├── sample1.msi.json
│   │       ├── diffs.txt
│   │       └── tumor.dist
│   ├── DnaRealignment
│   │   └── sample1
│   │       ├── sample1.bam
│   │       └── sample1.bam.bai
│   ├── VariantCaller
│   │   ├── sample1
│   │   │   ├── sample1.filtered.genome.vcf.gz
│   │   │   └── sample1.genome.vcf.gz
│   │   │   └── PiscesOptions.used.json
│   │       └── PsaraOptions.used.json
│   ├── SmallVariantFilter
│   │   └── sample1
│   │       ├── sample1_SmallVariants.genome.vcf.errorRates
│   │       └── sample1_SmallVariants.genome.vcf.gz
│   ├── CollapsedReads
│   │   └── sample1
│   │       ├── sample1_metrics.json
│   │       ├── sample1_R1.fastq.gz
│   │       └── sample1_R2.fastq.gz
│   ├── CnvCaller
│   │   └── sample1
│   │       ├── sample1_CopyNumberVariants.vcf.gz
│   │       ├── sample1_foldChange.tsv
│   │       ├── sample1_normalizedBinCount.tsv
│   │       ├── sample1_rawBinCount.tsv
│   │       └── Craft_6735_20231026_113343.txt
│   ├── SamplesheetValidation
|   |   └── sample1
│   │       └── 20231026_224827_SampleSheet.csv
│   ├── DnaAlignment
│   │   └── sample1
│   │       ├── sample1.bam
│   │       └── sample1.bam.bai
│   ├── Tmb
│   │   └── sample1
│   │       ├── sample1.tmb.json
│   │       └── sample1_TMB_Trace.tsv
│   ├── CombinedVariantOutput
│   │   └── sample1
│   │       └── sample1_CombinedVariantOutput.tsv
│   ├── SampleAnalysisResults
|   |   └── sample1
│   │       └── sample1_SampleAnalysisResults.json
│   ├── RnaAlignment
|   |   └── sample1
│   │       ├── starLoad.Aligned.out.sam
│   │       └── starUnload.Aligned.out.sam
│   ├── MetricsOutput
|   |   └── sample1
│   │       └── MetricsOutput.tsv
│   |── MergedAnnotation
│   │   └── sample1
│   |       ├── sample1_MergedVariants_Annotated.json.gz
│   |       └── sample1_MergedVariants_Annotated.json.gz.jsi
│   |── inputs.json
│   └── SampleSheet.csv
|
├── gather
│   ├── Logs_Intermediates
│   │   ├── CombinedVariantOutput
│   │   │   ├── sample1
│   │   │   │   └── sample1_CombinedVariantOutput.tsv
│   │   │   ├── sample1
│   │   │   │   └── sample1_CombinedVariantOutput.tsv
...
│   │   ├── Msi
│   │   │   ├── 125501523-23269S0032-23TSOD58-8471
│   │   │   │   └── 125501523-23269S0032-23TSOD58-8471.msi.json
│   │   │   ├── 125510260-23269S0002-23TSOD58-8471
│   │   │   │   └── 125510260-23269S0002-23TSOD58-8471.msi.json
...
│   │   ├── Contamination
│   │   │   ├── sample1
│   │   │   │   └── sample1.contamination.json
│   │   │   ├── sample1
...
│   │   ├── DnaQCMetrics
│   │   │   ├── sample1
│   │   │   │   ├── sample1.aligned.metrics.json
│   │   │   │   ├── sample1.cnv.metrics.json
│   │   │   │   ├── sample1.collapsed.metrics.json
│   │   │   │   └── sample1.stitched.metrics.json
...
│   │   ├── CollapsedReads
│   │   │   ├── sample1
│   │   │   │   └── sample1_metrics.json
...
│   │   ├── SampleAnalysisResults
│   │   │   ├── sample1_SampleAnalysisResults.json
...
│   │   ├── MetricsOutput
│   │   │   └── MetricsOutput.tsv
│   │   ├── Annotation
│   │   │   ├── sample1
│   │   │   │   └── sample1_SmallVariants_Annotated.json.gz
...
│   │   ├── MergedAnnotation
│   │   │   ├── sample1
│   │   │   │   └── sample1_MergedVariants_Annotated.json.gz
...
│   │   ├── RunQc
│   │   │   └── RunQCMetrics.json
│   │   └── Tmb
│   │       ├── sample1
│   │       │   ├── sample1.tmb.json
│   │       │   └── sample1_TMB_Trace.tsv
...
│   ├── Results
│   │   ├── sample1
│   │   │   ├── sample1_CombinedVariantOutput.tsv
│   │   │   ├── sample1_CopyNumberVariants.vcf
│   │   │   ├── sample1_MergedSmallVariants.genome.vcf
│   │   │   ├── sample1_MergedVariants_Annotated.json.gz
│   │   │   └── sample1_TMB_Trace.tsv
...
│   ├── inputs.json
│   └── SampleSheet.csv
├── demultiplexOutput
│   ├── Logs_Intermediates
│   │   ├── FastqGeneration
│   │   │   ├── Reports
│   │   │   │   ├── Lane_1
│   │   │   │   │   ├── Adapter_Metrics.csv
│   │   │   │   │   ├── Demultiplex_Stats.csv
│   │   │   │   │   ├── fastq_list.csv
│   │   │   │   │   ├── Index_Hopping_Counts.csv
│   │   │   │   │   ├── IndexMetricsOut.bin
│   │   │   │   │   ├── RunInfo.xml
│   │   │   │   │   ├── SampleSheet.csv
│   │   │   │   │   └── Top_Unknown_Barcodes.csv
│   │   │   │   └── Lane_2
│   │   │   │       ├── Adapter_Metrics.csv
│   │   │   │       ├── Demultiplex_Stats.csv
│   │   │   │       ├── fastq_list.csv
│   │   │   │       ├── Index_Hopping_Counts.csv
│   │   │   │       ├── IndexMetricsOut.bin
│   │   │   │       ├── RunInfo.xml
│   │   │   │       ├── SampleSheet.csv
│   │   │   │       └── Top_Unknown_Barcodes.csv
│   │   │   ├── Logs
│   │   │   │   ├── Lane_2
│   │   │   │   │   ├── Errors.log
│   │   │   │   │   ├── FastqComplete.txt
│   │   │   │   │   ├── Info.log
│   │   │   │   │   └── Warnings.log
│   │   │   │   └── Lane_1
│   │   │   │       ├── Errors.log
│   │   │   │       ├── FastqComplete.txt
│   │   │   │       ├── Info.log
│   │   │   │       └── Warnings.log
│   │   │   ├── sample1
│   │   │   │   ├── sample1_S2_L001_R1_001.fastq.gz
│   │   │   │   ├── sample1_S2_L001_R2_001.fastq.gz
│   │   │   │   ├── sample1_S2_L002_R1_001.fastq.gz
│   │   │   │   └── sample1_S2_L002_R2_001.fastq.gz
...
│   │   │   ├── dsdm.json
│   │   │   ├── FastqGeneration-150369-20231026-223016.stderr
│   │   │   ├── FastqGeneration-150369-20231026-223016.stdout
│   │   │   ├── FastqGeneration-20231026222542.log
│   │   │   ├── FastqGeneration-417-20231026-222542.stderr
│   │   │   ├── FastqGeneration-417-20231026-222542.stdout
│   │   │   └── SampleSheet_combined.csv
│   │   ├── SamplesheetValidation
│   │   │   ├── 20231026_222415_SampleSheet.csv
│   │   │   ├── dsdm.json
│   │   │   └── SamplesheetValidation-20231026222415.log
│   │   ├── RunQc
│   │   │   ├── dsdm.json
│   │   │   ├── RunQc-20231026222537.log
│   │   │   └── RunQCMetrics.json
│   │   └── ResourceVerification
│   │       ├── dsdm.json
│   │       └── ResourceVerification-20231026222421.log
│   ├── demultiplex_cromwell_executions.tar.gz
│   ├── inputs.json
│   ├── receipt
│   ├── SampleSheet.csv
│   └── trusight-oncology-500-ruo_ruo-2.2.0.12_20231026-232354.log
├── logs
│   ├── sample1_cromwell_executions.tar.gz
│   ├── sample1_scatter_logs.tar.gz
...
│   ├── analysis_results_logs.tar.gz
│   └── gather_cromwell_executions.tar.gz
└── MetricsOutput.tsv

Notes

• Samplesheet input is optional, if not specified the analysis app looks for the samplesheet in top level of runfolder
• When running in scatter / gather mode, demultiplexing via the local app must always be first performed (i.e it can't be started from previously demultiplexed data and reusing fastqs). To achieve the equivalent, -iupload_demultiplex_output=false may be specified to not upload the output of demultiplexing from the job. When used in conjunction with -iinclude_samples="sample1", this would effectively just run and output data for the given sample(s) as if the local app were just run from fastqs for a single sample
• Jobs are launched per sample parsed from the 'Pair_ID' column, this means if the samplesheet is formatted for running in paired analysis mode, the fastqs for all samples for the given pair ID will be used for the analysis
• sample names specified to -iinclude_samples or -iexclude_samples should be specified as given in the Pair_ID column of the samplesheet, or as regex pattern(s) which will be matched against the Pair_ID column
• Intermediary genome vcfs found in scatter/ and gather/Logs_Intermediates/ are compressed before uploading to save on storage
• All log files are gathered up before uploading and combined into tar files, there is one tar file per file from the scatter step and one from the gather step. This is to reduce the total number of files uploaded at the end.

This app was created by East Genomics GLH