Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add MMseqs2 clustering and taxonomy #6574
base: main
Are you sure you want to change the base?
Add MMseqs2 clustering and taxonomy #6574
Changes from 61 commits
6467853
77d1bef
3ad2ea4
b9da697
0ac3019
75f154d
fc600b6
403a2d9
6c308bb
afd1577
ec54759
d48ab99
71779aa
5f584df
0c250c5
f96947c
09771c7
7f2c49e
88da237
157360f
410d4f4
3c3d188
8fbf055
b82acf0
0f859b5
0bd7ef7
e1299a9
76da478
d0fac9f
279f796
cb26bf4
aafcdfe
2333a5a
79f3410
dd5ab01
cbf00be
29e9015
1d28b66
53c1550
6649879
8bac29c
784bb52
5f3a7e3
98e72f3
eb4187d
1ba4a1f
f46798e
65c07c3
3efd6ab
78ba0c6
220cd5a
c729ba6
072fc28
54a432f
fe0b983
3637da0
6d07b87
940613d
09e0d94
41cb56d
2e2364d
816451e
dde715a
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing that I do not like about this is that most of the downloaded databases are not properly versioned. For instance GTDB is downloaded from latest: https://github.com/soedinglab/MMseqs2/blob/c2c3ad9c2956fac691d5a6041a9a4affa7fa27ad/data/workflow/databases.sh#L148
So we can not guarantee that gtdb on one Galaxy is the same as gtdb on another Galaxy.
That's is not your fault, but upstream. I would suggest to ask upstream if they could provide versioned downloads.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe it would be better to have a data manager that takes fasta as input and calls
mmseqs createdb
like hereL https://github.com/soedinglab/MMseqs2/blob/c2c3ad9c2956fac691d5a6041a9a4affa7fa27ad/data/workflow/databases.sh#L388fasta could be taken from other data tables .. but its difficult, because it will be multiple data bases.
Or is the fasta removed before its added to the data table, they call
rmdb
....Main question for me is: what is actually stored in the output folder and how big is it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes I agree with you about db versions, do I need to create an issue on MMseqs repo to ask them ?
If I understand correctly, the idea would be to take the fasta files already existing in Galaxy from the databases proposed by mmseqs and use
createdb
to create a database that can be used in the command suite without having to download themmseqs databases
. I wonder how easy it is to find out which fasta files are used to buildmmseqs databases
?The output folder of
createdb
command has the same composition as after ammseqs databases
(you can find an example in Swiss-prot directory in test files). There is a text file with sequences representing the database (not a fasta format), index files and files containing general information (lookup file, identifiers assigned by MMseqs2 and correspondance with original sequences).For test file which is 568K, the createdb output directory is 836K (don't know if it can be useful such a small file)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would be perfect.
This would be my idea.
This would be good anyway, otherwise we can not answer this question to users.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are these databases that are downloaded (fasta, some index, or something else)? For instance: is gtdb the full gtdb?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm pretty sure that gtdb mmseqs database is not the full gtdb. I'm downloading it with
mmseqs databases
and they tell me 66GB while full gtdb release220 is 107GB.In addition, the output format will not be the same, as
mmseqs databases
gives a very particular form to its directory and has its own files (as explained withcreatedb
in the previous comment: txt file with sequence representing the database, index files, general information files).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK. So one thing that we should probably take case of (if we use fasta from other data tables) is that the mmseqs files are installed to a separate folder.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't have to be in a separate folder, but I prefer to work that way as several files make up the output.
After checking, the GTDB database using mmseqs is finally 170GB, with the file containing the sequences being 100GB.