Added a more comprehensive diagram of the flow

README.md

@@ -6,6 +6,12 @@ The pipeline is visualized in the following scheme:
 
 <img src="https://github.com/Easybel/DetectionGV/blob/main/Pipeline_WGS.png" width="900">
 
+A more detailed diagram can be found on LucidChart <https://lucid.app/documents/view/fa9f52da-edf4-4fb2-9b6c-d01996f4768e> or under [/documentation](/documentation)
+
+Preview:
+
+<img src="https://github.com/Easybel/DetectionGV/blob/main/documentation/DetectionGV_preview.png" width="900">
+
 # Overview of folders and scripts
 - **[0_WGSPipeline](https://github.com/Easybel/DetectionGV/tree/main/0_WGSPipeline)**  
   - This contains all bioinformatic scripts needed to analyse whole genome, raw sequencing reads of bacterial transformation hybrids.

documentation/DetectionGV.csv

@@ -0,0 +1,209 @@
+Id,Name,Shape Library,Page ID,Contained By,Group,Line Source,Line Destination,Source Arrow,Destination Arrow,Status,Text Area 1,Text Area 2,Text Area 3,Text Area 4,Text Area 5,Text Area 6,Text Area 7,Comments
+1,Document,,,,,,,,,Draft,DetectionGV,,,,,,,
+2,Page,,,,,,,,,,Page 1,,,,,,,
+3,Cloud,Geometric Shapes,2,,,,,,,,3_CombiScriptInsertions.sh,,,,,,,
+4,Cloud,Geometric Shapes,2,,,,,,,,2_CombiScriptSamtools.sh,,,,,,,
+5,Cloud,Geometric Shapes,2,,,,,,,,1_QCScript.sh,,,,,,,
+6,Cloud,Geometric Shapes,2,,,,,,,,0_DicIndex.sh,,,,,,,
+7,DividerBlock,,2,,,,,,,,,,,,,,,
+8,Decision,Flowchart Shapes/Containers,2,,,,,,,,.fasta    (recipient WT),,,,,,,
+9,Text,Standard,2,,,,,,,,Trimmomatic trimmomatic PE,,,,,,,
+10,Decision,Flowchart Shapes/Containers,2,,,,,,,,raw reads .fastq(.gz) of hybrids,,,,,,,
+11,Text,Standard,2,,,,,,,,Align reads to reference genome (recipient wild type) bwa mem -B 1 -O 1,,,,,,,
+12,Text,Standard,2,,,,,,,,FastQC fastqc,,,,,,,
+13,Decision,Flowchart Shapes/Containers,2,,,,,,,,quality reports .zip/.html,,,,,,,
+14,Text,Standard,2,,,,,,,,pile up bcftools mpileup,,,,,,,
+15,Decision,Flowchart Shapes/Containers,2,,,,,,,,_2Ref_bcf.vcf,,,,,,,
+16,Text,Standard,2,,,,,,,,calculate coverage with BedTools genomeCoverageBed,,,,,,,
+17,Decision,Flowchart Shapes/Containers,2,,,,,,,,coverage _2Ref_coverage.txt,,,,,,,
+18,Text,Standard,2,,,,,,,,"""Hard"" align reads to donor genome (recipient wild type) bwa mem",,,,,,,
+19,Decision,Flowchart Shapes/Containers,2,,,,,,,,_2Donor.sam,,,,,,,
+20,Text,Standard,2,,,,,,,,sort samtools sort,,,,,,,
+21,Decision,Flowchart Shapes/Containers,2,,,,,,,,_2Donor_sort.bam,,,,,,,
+22,Text,Standard,2,,,,,,,,index samtools index,,,,,,,
+23,Decision,Flowchart Shapes/Containers,2,,,,,,,,_2Donor_sort.bam.bai,,,,,,,
+24,Text,Standard,2,,,,,,,,calculate coverage with BedTools genomeCoverageBed,,,,,,,
+25,Decision,Flowchart Shapes/Containers,2,,,,,,,,coverage _2Donor_coverage.txt,,,,,,,
+26,Text,Standard,2,,,,,,,,Variant calling bcftools call,,,,,,,
+27,Decision,Flowchart Shapes/Containers,2,,,,,,,,variant list _2Ref_bcfcall.vcf,,,,,,,
+28,Display,Flowchart Shapes/Containers,2,,,,,,,,"paths, filenames, sampleType, filter parameters",,,,,,,
+29,Decision,Flowchart Shapes/Containers,2,,,,,,,,SNPSummary.mat,,,,,,,
+30,Text,Standard,2,,,,,,,,A0c_AccessoryGenome.m,,,,,,,
+31,Display,Flowchart Shapes/Containers,2,,,,,,,,"paths, filenames, filter parameters, genome length recipient (.fasta letter count AGTC)",,,,,,,
+32,Sticky note,Sticky Notes,2,,,,,,,,list of single replacements + some metadata,,,,,,,
+33,Sticky note,Sticky Notes,2,,,,,,,,filters single replacements,,,,,,,
+34,Sticky note,Sticky Notes,2,,,,,,,,list of single replacements + their positions -------------------  for each sample,,,,,,,
+35,Sticky note,Sticky Notes,2,,,,,,,,filters ???,,,,,,,
+36,Sticky note,Sticky Notes,2,,,,,,,,list of single replacements + their positions -------------------  for each sample,,,,,,,
+37,Cloud,Geometric Shapes,2,,,,,,,,2_CombiScript Samtools.sh,,,,,,,
+38,Decision,Flowchart Shapes/Containers,2,,,,,,,,.dict (recipient WT),,,,,,,
+39,Cloud,Geometric Shapes,2,,,,,,,,1_QCScript.sh,,,,,,,
+40,Decision,Flowchart Shapes/Containers,2,,,,,,,,raw reads .fastq(.gz) of donor,,,,,,,
+41,Decision,Flowchart Shapes/Containers,2,,,,,,,,(un)paired reads          _P.fastq          _U.fastq,,,,,,,
+42,Decision,Flowchart Shapes/Containers,2,,,,,,,,variant list _2Ref_bcf.vcf,,,,,,,
+43,Delete,Salesforce Architecture Diagrams,2,,,,,,,,,,,,,,,
+44,Decision,Flowchart Shapes/Containers,2,,,,,,,,.fasta        (donor),,,,,,,
+45,LinkUnfurlBlock,,2,,,,,,,,Basic Local Alignment Search Tool,Nih,,,,,,
+46,Text,Standard,2,,,,,,,,A0d_Multimapper.m,,,,,,,
+47,Text,Standard,2,,,,,,,,2x,,,,,,,
+48,Decision,Flowchart Shapes/Containers,2,,,,,,,,.fasta        (recipient WT),,,,,,,
+49,Decision,Flowchart Shapes/Containers,2,,,,,,,,.fasta        (recipient WT),,,,,,,
+50,Decision,Flowchart Shapes/Containers,2,,,,,,,,.fasta        (recipient WT),,,,,,,
+51,Decision,Flowchart Shapes/Containers,2,,,,,,,,.fasta        (donor),,,,,,,
+52,Decision,Flowchart Shapes/Containers,2,,,,,,,,hittable (text),,,,,,,
+53,Decision,Flowchart Shapes/Containers,2,,,,,,,,hitttable (text),,,,,,,
+54,Right arrow,Geometric Shapes,2,,,,,,,,"""simulates"" hybrid with 100% transfer",,,,,,,
+55,Decision,Flowchart Shapes/Containers,2,,,,,,,,mm_list.txt,,,,,,,
+56,Decision,Flowchart Shapes/Containers,2,,,,,,,,mm.txt,,,,,,,
+57,Text,Standard,2,,,,,,,,A0b_MasterListFiltering.m,,,,,,,
+58,Decision,Flowchart Shapes/Containers,2,,,,,,,,acc..._allBP.txt,,,,,,,
+59,Decision,Flowchart Shapes/Containers,2,,,,,,,,acc..._allBP.mat,,,,,,,
+60,Decision,Flowchart Shapes/Containers,2,,,,,,,,master list ml....mat,,,,,,,
+61,Decision,Flowchart Shapes/Containers,2,,,,,,,,master list  ml...txt,,,,,,,
+62,Text,Standard,2,,,,,,,,A1_SNP2CNP.m,,,,,,,
+63,Decision,Flowchart Shapes/Containers,2,,,,,,,,CNPSummary.mat,,,,,,,
+64,Cloud,Geometric Shapes,2,,,,,,,,2_CombiScript Samtools.sh,,,,,,,
+65,Decision,Flowchart Shapes/Containers,2,,,,,,,,.dict (recipient WT),,,,,,,
+66,Cloud,Geometric Shapes,2,,,,,,,,1_QCScript.sh,,,,,,,
+67,Decision,Flowchart Shapes/Containers,2,,,,,,,,raw reads .fastq(.gz) of recipient,,,,,,,
+68,Decision,Flowchart Shapes/Containers,2,,,,,,,,(un)paired reads          _P.fastq          _U.fastq,,,,,,,
+69,Decision,Flowchart Shapes/Containers,2,,,,,,,,SNP artefacts ReciWT_2Ref_ bcfcall.vcf,,,,,,,
+70,Right arrow,Geometric Shapes,2,,,,,,,,"""simulates"" hybrid with 0% transfer",,,,,,,
+71,DividerBlock,,2,,,,,,,,,,,,,,,
+72,Text,Standard,2,,,,,,,,Create dictionary picard.jar CreateSequenceDictionary,,,,,,,
+73,DividerBlock,,2,,,,,,,,,,,,,,,
+74,Text,Standard,2,,,,,,,,2x,,,,,,,
+75,Text,Standard,2,,,,,,,,A0_VariantFiltering.m,,,,,,,
+76,Decision,Flowchart Shapes/Containers,2,,,,,,,,SNP artefact list _ArteSNPs.mat,,,,,,,
+77,Delete,Salesforce Architecture Diagrams,2,,204,,,,,,,,,,,,,
+78,Text,Standard,2,,204,,,,,,"bcf, sam, bam",,,,,,,
+79,Delete,Salesforce Architecture Diagrams,2,,205,,,,,,,,,,,,,
+80,Text,Standard,2,,205,,,,,,"bcfcall, sam, bam, txt",,,,,,,
+81,Decision,Flowchart Shapes/Containers,2,,,,,,,,(un)paired reads        _P.fastq _U.fastq,,,,,,,
+82,Text,Standard,2,,,,,,,,"A3_Cov2DelDup.m
+finds deletions + duplications     based on coverage",,,,,,,
+83,Sticky note,Sticky Notes,2,,,,,,,,how frequently a certain locus was read,,,,,,,
+84,Decision,Flowchart Shapes/Containers,2,,206,,,,,,,,,,,,,
+85,Decision,Flowchart Shapes/Containers,2,,206,,,,,,,,,,,,,
+86,Decision,Flowchart Shapes/Containers,2,,206,,,,,,(un)paired reads        _P.fastq _U.fastq,,,,,,,
+87,Decision,Flowchart Shapes/Containers,2,,,,,,,,artefacts _ArteCov.txt,,,,,,,
+88,Decision,Flowchart Shapes/Containers,2,,,,,,,,_DelDup _sample.mat,,,,,,,
+89,Text,Standard,2,,,,,,,,"A4_Cov2Ins.m
+finds insertions based on coverage",,,,,,,
+90,Decision,Flowchart Shapes/Containers,2,,,,,,,,AccMM2Genes.mat,,,,,,,
+91,Sticky note,Sticky Notes,2,,,,,,,,"Load existing .mat and empty all cells (empty_AccMM2Genes.mat). Run A2_List2Genes.m in txtListStartEnd-mode with acc..._allBP.txt as far as possible, skip/fix possible errors until MultiHitStat is generated. Save as .mat and filter for FracMean > 0 (e.g. use FilterStructArray_by_PH.m). ",,,,,,,
+92,Text,Standard,2,,,,,,,,A2_List2Genes.m,,,,,,,
+93,Decision,Flowchart Shapes/Containers,2,,,,,,,,bed file ...bed.mat,,,,,,,
+94,Cloud Deployment Manager,Google Cloud Platform 2021,2,,,,,,,,plots,,,,,,,
+95,Decision,Flowchart Shapes/Containers,2,,,,,,,,.dict (recipient WT),,,,,,,
+96,Decision,Flowchart Shapes/Containers,2,,,,,,,,.dict (donor),,,,,,,
+97,Decision,Flowchart Shapes/Containers,2,,,,,,,,_2Ref.sam,,,,,,,
+98,Text,Standard,2,,,,,,,,sort samtools sort,,,,,,,
+99,Decision,Flowchart Shapes/Containers,2,,,,,,,,_2Ref_sort.bam,,,,,,,
+100,Text,Standard,2,,,,,,,,index samtools index,,,,,,,
+101,Decision,Flowchart Shapes/Containers,2,,,,,,,,_2Ref_sort.bam.bai,,,,,,,
+102,Display,Flowchart Shapes/Containers,2,,,,,,,,"paths, filenames, filter parameters, genome length recipient (.fasta letter count AGTC)",,,,,,,
+103,Cloud Deployment Manager,Google Cloud Platform 2021,2,,,,,,,,plots,,,,,,,
+104,Display,Flowchart Shapes/Containers,2,,,,,,,,"paths, filenames, filter parameters, genome length recipient (.fasta letter count AGTC)",,,,,,,
+105,Diagram key,Standard,2,,,,,,,,Legend,recipient file,donor file,reciprocal hybrid (recipient/donor) file,hybrid (donor/recipient) file,Matlab program (.m) or bash command,bash file,
+106,Line,,2,,,8,72,None,Arrow,,,,,,,,,
+107,Line,,2,,,72,95,None,Arrow,,,,,,,,,
+108,Line,,2,,,10,9,None,Arrow,,,,,,,,,
+109,Line,,2,,,9,86,None,Arrow,,,,,,,,,
+110,Line,,2,,,10,12,None,Arrow,,,,,,,,,
+111,Line,,2,,,12,13,None,Arrow,,,,,,,,,
+112,Line,,2,,,86,12,None,Arrow,,,,,,,,,
+113,Line,,2,,,95,11,None,Arrow,,,,,,,,,
+114,Line,,2,,,11,97,None,Arrow,,,,,,,,,
+115,Line,,2,,,98,99,None,Arrow,,,,,,,,,
+116,Line,,2,,,100,101,None,Arrow,,,,,,,,,
+117,Line,,2,,,97,98,None,Arrow,,,,,,,,,
+118,Line,,2,,,99,100,None,Arrow,,,,,,,,,
+119,Line,,2,,,14,15,None,Arrow,,,,,,,,,
+120,Line,,2,,,16,17,None,Arrow,,,,,,,,,
+121,Line,,2,,,95,14,None,Arrow,,,,,,,,,
+122,Line,,2,,,86,18,None,Arrow,,,,,,,,,
+123,Line,,2,,,18,19,None,Arrow,,,,,,,,,
+124,Line,,2,,,20,21,None,Arrow,,,,,,,,,
+125,Line,,2,,,22,23,None,Arrow,,,,,,,,,
+126,Line,,2,,,19,20,None,Arrow,,,,,,,,,
+127,Line,,2,,,21,22,None,Arrow,,,,,,,,,
+128,Line,,2,,,24,25,None,Arrow,,,,,,,,,
+129,Line,,2,,,72,96,None,Arrow,,,,,,,,,
+130,Line,,2,,,96,18,None,Arrow,,,,,,,,,
+131,Line,,2,,,95,98,None,Arrow,,,,,,,,,
+132,Line,,2,,,96,20,None,Arrow,,,,,,,,,
+133,Line,,2,,,21,24,None,Arrow,,,,,,,,,
+134,Line,,2,,,26,27,None,Arrow,,,,,,,,,
+135,Line,,2,,,15,26,None,Arrow,,,,,,,,,
+136,Line,,2,,,99,16,None,Arrow,,,,,,,,,
+137,Line,,2,,,99,14,None,Arrow,,,,,,,,,
+138,Line,,2,,,86,11,None,Arrow,,,,,,,,,
+139,Line,,2,,,28,75,None,Arrow,,,,,,,,,
+140,Line,,2,,,27,75,None,Arrow,,,,,,,,,
+141,Line,,2,,,75,29,None,Arrow,,,,,,,,,
+142,Line,,2,,,31,30,None,Arrow,,,,,,,,,
+143,Line,,2,,,,,None,None,,,,,,,,,
+144,Line,,2,,,40,39,None,Arrow,,,,,,,,,
+145,Line,,2,,,39,41,None,Arrow,,,,,,,,,
+146,Line,,2,,,41,37,None,Arrow,,,,,,,,,
+147,Line,,2,,,38,37,None,Arrow,,,,,,,,,
+148,Line,,2,,,37,42,None,Arrow,,,,,,,,,
+149,Line,,2,,,42,30,None,Arrow,,,,,,,,,
+150,Line,,2,,,21,43,None,Arrow,,,,,,,,,
+151,Line,,2,,,99,43,None,Arrow,,,,,,,,,
+152,Line,,2,,,97,43,None,Arrow,,,,,,,,,
+153,Line,,2,,,19,43,None,Arrow,,,,,,,,,
+154,Line,,2,,,44,72,None,Arrow,,,,,,,,,
+155,Line,,2,,,45,52,None,Arrow,,,,,,,,,
+156,Line,,2,,,45,53,None,Arrow,,,,,,,,,
+157,Line,,2,,,48,45,None,Arrow,,subject,,,,,,,
+158,Line,,2,,,49,45,None,Arrow,,query,,,,,,,
+159,Line,,2,,,50,45,None,Arrow,,subject,,,,,,,
+160,Line,,2,,,51,45,None,Arrow,,query,,,,,,,
+161,Line,,2,,,53,46,None,Arrow,,,,,,,,,
+162,Line,,2,,,52,46,None,Arrow,,,,,,,,,
+163,Line,,2,,,46,56,None,Arrow,,,,,,,,,
+164,Line,,2,,,46,55,None,Arrow,,,,,,,,,
+165,Line,,2,,,30,58,None,Arrow,,,,,,,,,
+166,Line,,2,,,30,59,None,Arrow,,,,,,,,,
+167,Line,,2,,,57,60,None,Arrow,,,,,,,,,
+168,Line,,2,,,56,62,None,Arrow,,,,,,,,,
+169,Line,,2,,,42,57,None,Arrow,,,,,,,,,
+170,Line,,2,,,62,63,None,Arrow,,,,,,,,,
+171,Line,,2,,,29,62,None,Arrow,,,,,,,,,
+172,Line,,2,,,67,66,None,Arrow,,,,,,,,,
+173,Line,,2,,,66,68,None,Arrow,,,,,,,,,
+174,Line,,2,,,68,64,None,Arrow,,,,,,,,,
+175,Line,,2,,,65,64,None,Arrow,,,,,,,,,
+176,Line,,2,,,64,69,None,Arrow,,,,,,,,,
+177,Line,,2,,,57,61,None,Arrow,,,,,,,,,
+178,Line,,2,,,69,75,None,Arrow,,,,,,,,,
+179,Line,,2,,,75,76,None,Arrow,,,,,,,,,
+180,Line,,2,,,64,,None,Arrow,,,,,,,,,
+181,Line,,2,,,37,,None,Arrow,,,,,,,,,
+182,Line,,2,,,81,75,None,Arrow,,,,,,,,,
+183,Line,,2,,,81,75,None,Arrow,,,,,,,,,
+184,Line,,2,,,69,62,None,Arrow,,,,,,,,,
+185,Line,,2,,,87,82,None,Arrow,,,,,,,,,
+186,Line,,2,,,82,88,None,Arrow,,,,,,,,,
+187,Line,,2,,,91,90,None,Arrow,,,,,,,,,
+188,Line,,2,,,58,91,None,Arrow,,,,,,,,,
+189,Line,,2,,,63,92,None,Arrow,,,,,,,,,
+190,Line,,2,,,90,92,None,Arrow,,,,,,,,,
+191,Line,,2,,,58,62,None,Arrow,,,,,,,,,
+192,Line,,2,,,61,62,None,Arrow,,,,,,,,,
+193,Line,,2,,,17,82,None,Arrow,,,,,,,,,
+194,Line,,2,,,93,92,None,Arrow,,,,,,,,,
+195,Line,,2,,,92,94,None,Arrow,,,,,,,,,
+196,Line,,2,,,86,81,None,None,,,,,,,,,
+197,Line,,2,,,95,38,None,None,,,,,,,,,
+198,Line,,2,,,102,82,None,Arrow,,,,,,,,,
+199,Line,,2,,,25,89,None,Arrow,,,,,,,,,
+200,Line,,2,,,82,103,None,Arrow,,,,,,,,,
+201,Line,,2,,,58,89,None,Arrow,,???,,,,,,,
+202,Line,,2,,,104,89,None,Arrow,,,,,,,,,
+203,Line,,2,,,95,65,None,None,,,,,,,,,
+204,Group 1,,2,,,,,,,,,,,,,,,
+205,Group 2,,2,,,,,,,,,,,,,,,
+206,Group 3,,2,,,,,,,,,,,,,,,

documentation/README.md

@@ -0,0 +1,4 @@
+The diagram was created with LucidChart. Unfortunately this is closed and some featurs are premium only, so it is a kind of vendor lock-in. I can share the diagram to your LucidChart account, otherwise feel free to rebuild it from the given files.
+<https://lucid.app/documents/view/fa9f52da-edf4-4fb2-9b6c-d01996f4768e>
+
+ <p xmlns:cc="http://creativecommons.org/ns#" >This diagram is marked with <a href="http://creativecommons.org/publicdomain/zero/1.0?ref=chooser-v1" target="_blank" rel="license noopener noreferrer" style="display:inline-block;">CC0 1.0<img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1"><img style="height:22px!important;margin-left:3px;vertical-align:text-bottom;" src="https://mirrors.creativecommons.org/presskit/icons/zero.svg?ref=chooser-v1"></a> and dedicated to the public domain.</p>