Abyss assembler issue: Missing files in temp directory

[rafelafrance]

@zgliwen

Let's move your issue over here. please.

=====================================================================================================
@rafelafrance
hi,

I also had the same issue. The difference from others was that I used Abyss rather than Trinity as the assembler, and it reported the similar error:
ERROR: Exception: [Errno 2] No such file or directory: '/data/results/atram_out_220407/kaM_49223_imd/atram_qfz_iu_n/kaM_49223_seq.fa_01_xdlazpp6/output.fasta-unitigs.fa'

My command was:
$ atram.py --database atram_out_220407/kaM_49223 --query basic_info/kaM_seq.fa --assembler abyss -t /data/results/atram_out_220407/kaM_49223_imd --keep-temp-dir --no-filter --path miniconda3/envs/py3/bin -o /data/results/atram_out_220407/kaM_49223

However, with the fasta files created by atram in the temporary directory, I can get the assemble by Abyss-pe, and the command was
$ abyss-pe np=8 name=/data/results/atram_out_220407/kaM_49223_abyss k=64 in='atram_out_220407/kaM_49223_imd/atram_qfz_iu_n/kaM_49223_seq.fa_01_xdlazpp6/paired_1.fasta atram_out_220407/kaM_49223_imd/atram_qfz_iu_n/kaM_49223_seq.fa_01_xdlazpp6/paired_2.fasta'

The version of softwares I used were:
aTRAM 2.4.3
abyss 2.3.4-h41cdee2_1

Could you give me some suggestions, please? Thanks for your help.

[zgliwen]

Actually, I used Abyss as the assembler, not Trinity ...

[rafelafrance]

Noted. Thank you.

[rafelafrance]

@juliema I'm having trouble replicating this issue here. Have you run into this problem when working on the "Pronghorn" server?

[emilyostrow]

I ran into this problem when I forgot to activate my conda environment where Abyss was installed. Activating the environment fixed it. I think the program can't find abyss.

[zgliwen]

I ran into this problem when I forgot to activate my conda environment where Abyss was installed. Activating the environment fixed it. I think the program can't find abyss.

@emilyostrow Thanks. I actually ran aTRAM in the conda environment includes Abyss, but failed. I think you are right that aTRAM could not find Abyss in my environment, but I don't know why. Now I could get the intermediate files from aTRAM and assemble sequences with Abyss.

[juliema]

you could put in the path to abyss in the command. I think it is --path <path to abyss> that way it will find it.

…

On Fri, Jun 17, 2022 at 10:03 PM zgliwen ***@***.***> wrote: I ran into this problem when I forgot to activate my conda environment where Abyss was installed. Activating the environment fixed it. I think the program can't find abyss. Thanks. I actually ran aTRAM in the conda environment includes Abyss, but failed. I think you are right that aTRAM could not find Abyss in my environment, but I don't know why. Now I could get the intermediate files from aTRAM and assemble sequences with Abyss. — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AZ3RHNJD4NWCRAO64BF3VPVKDPANCNFSM5SY6AWBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zgliwen]

you could put in the path to abyss in the command. I think it is --path that way it will find it.

@juliema Thanks, Juliema. I tried it, but failed again. The commands I used and the whole log file were shown below. Hope this information will be useful. Thank you.

The old command:
$ atram.py --database atram_out_220407/kaM_49223 --query basic_info/kaM_seq.fa --assembler abyss -t /data/results/atram_out_220407/kaM_49223_imd --keep-temp-dir --no-filter --path miniconda3/envs/py3/bin -o /data/results/atram_out_220407/kaM_49223

The new command:
$ /data/software/aTRAM/atram.py --database results/atram_out_220407/kaM_49223 --query /data/basic_info/kaM_seq.fa --assembler abyss -t results/atram_out_220407/kaM_49223_imd --keep-temp-dir --abyss-kmer 30 --no-filter --path /data/miniconda3/envs/py3/bin/abyss-pe -o results/atram_out_220407/kaM_49223_test > results/atram_out_220407/kaM_49223_test.log 2>&1

$ less results/atram_out_220407/kaM_49223_test.log
( 617211) 2022-06-24 19:47:12.072941 INFO : ################################################################################
( 617211) 2022-06-24 19:47:12.073074 INFO : aTRAM version: v2.4.3
( 617211) 2022-06-24 19:47:12.073095 INFO : Python version: 3.9.2 (default, Mar 3 2021, 20:02:32) [GCC 7.3.0]
( 617211) 2022-06-24 19:47:12.073110 INFO : /data/software/aTRAM/atram.py --database results/atram_out_220407/kaM_49223 --query /data/basic_info/kaM_seq.fa --assembler abyss -t results/atram_out_220407/kaM_49223_imd --keep-temp-dir --abyss-kmer 30 --no-filter --path /data/miniconda3/envs/py3/bin/abyss-pe -o results/atram_out_220407/kaM_49223_test
( 617211) 2022-06-24 19:47:12.073195 INFO : aTRAM blast DB = " atram_out_220407/kaM_49223", query = "kaM_seq.fa", iteration 1
( 617211) 2022-06-24 19:47:12.073490 INFO : Blasting query against shards: iteration 1
( 617211) 2022-06-24 19:47:24.697196 INFO : All 8 blast results completed
( 617211) 2022-06-24 19:47:24.697776 INFO : 1028 blast hits in iteration 1
( 617211) 2022-06-24 19:47:24.697814 INFO : Writing assembler input files: iteration 1
( 617211) 2022-06-24 19:47:31.853828 INFO : Assembling shards with abyss: iteration 1
( 617211) 2022-06-24 19:47:32.022082 ERROR: Exception: [Errno 2] No such file or directory: '/data/results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_ seq.fa_01_qyk7lwxp/output.fasta-unitigs.fa'
( 617211) 2022-06-24 19:47:32.022342 INFO : 0 total contigs after iteration 1

[juliema]

ok two things. I would remove the abyss-pe from the path (if that is just the name of the program.. if it is a folder name then leave it) Can you see if this file exists? /data/results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_ seq.fa_01_qyk7lwxp/output.fasta-unitigs.fa -- if it does then the program is not finding it, if it does not exist -- which is probably the case, then there are two potential issues 1. the assembler is not working or 2. no contigs are being assembled. what locus are you assembling and do you have an idea of the coverage? just checking that you have the right version of abyss and it is working when you run it independently? If you could just try to run the abyss-pe command separately to make sure? happy to jump on zoom if you need to walk through it.

…

On Fri, Jun 24, 2022 at 5:29 AM zgliwen ***@***.***> wrote: you could put in the path to abyss in the command. I think it is --path that way it will find it. @juliema <https://github.com/juliema> Thanks, Juliema. I tried it, but failed again. The commands I used and the whole log file were shown below. Hope this information will be useful. Thank you. The old command: $ atram.py --database atram_out_220407/kaM_49223 --query basic_info/kaM_seq.fa --assembler abyss -t /data/results/atram_out_220407/kaM_49223_imd --keep-temp-dir --no-filter *--path miniconda3/envs/py3/bin* -o /data/results/atram_out_220407/kaM_49223 The new command: $ /data/software/aTRAM/atram.py --database results/atram_out_220407/kaM_49223 --query /data/basic_info/kaM_seq.fa --assembler abyss -t results/atram_out_220407/kaM_49223_imd --keep-temp-dir --abyss-kmer 30 --no-filter *--path /data/miniconda3/envs/py3/bin/abyss-pe* -o results/atram_out_220407/kaM_49223_test > results/atram_out_220407/kaM_49223_test.log 2>&1 $ less results/atram_out_220407/kaM_49223_test.log ( 617211) 2022-06-24 19:47:12.072941 INFO : ################################################################################ ( 617211) 2022-06-24 19:47:12.073074 INFO : aTRAM version: v2.4.3 ( 617211) 2022-06-24 19:47:12.073095 INFO : Python version: 3.9.2 (default, Mar 3 2021, 20:02:32) [GCC 7.3.0] ( 617211) 2022-06-24 19:47:12.073110 INFO : /data/software/aTRAM/atram.py --database results/atram_out_220407/kaM_49223 --query /data/basic_info/kaM_seq.fa --assembler abyss -t results/atram_out_220407/kaM_49223_imd --keep-temp-dir --abyss-kmer 30 --no-filter --path /data/miniconda3/envs/py3/bin/abyss-pe -o results/atram_out_220407/kaM_49223_test ( 617211) 2022-06-24 19:47:12.073195 INFO : aTRAM blast DB = " atram_out_220407/kaM_49223", query = "kaM_seq.fa", iteration 1 ( 617211) 2022-06-24 19:47:12.073490 INFO : Blasting query against shards: iteration 1 ( 617211) 2022-06-24 19:47:24.697196 INFO : All 8 blast results completed ( 617211) 2022-06-24 19:47:24.697776 INFO : 1028 blast hits in iteration 1 ( 617211) 2022-06-24 19:47:24.697814 INFO : Writing assembler input files: iteration 1 ( 617211) 2022-06-24 19:47:31.853828 INFO : Assembling shards with abyss: iteration 1 ( 617211) 2022-06-24 19:47:32.022082 ERROR: Exception: [Errno 2] No such file or directory: '/data/results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_ seq.fa_01_qyk7lwxp/output.fasta-unitigs.fa' ( 617211) 2022-06-24 19:47:32.022342 INFO : 0 total contigs after iteration 1 — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AZ3W4AWVQF7257ZFUYSTVQWSZBANCNFSM5SY6AWBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zgliwen]

@juliema Hi, Juliema.

There is no “output.fasta-unitigs.fa” file in the directory /data/results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_seq.fa_01_qyk7lwxp,
but exist several json files and five fasta files.

I can successfully assemble the paired reads into a complete contig, and the command I used is:
$ abyss-pe np=8 name=/data/results/atram_out_220407/kaM_49223_abyss k=64 in='results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_seq.fa_01_qyk7lwxp/paired_1.fasta results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_seq.fa_01_qyk7lwxp/paired_2.fasta'

I found that when we set the output file name in abyss, the path must be absolute. Maybe you had noticed it when writing aTRAM?

Thanks.

[juliema]

this is pretty common to need to go with absolute paths when running programs that call other programs. generally it is safest to always go with absolute paths. did this solve the problem for you?

…

On Sun, Jul 3, 2022 at 1:05 AM zgliwen ***@***.***> wrote: @juliema <https://github.com/juliema> Hi, Juliema. There is no “output.fasta-unitigs.fa” file in the directory /data/results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_seq.fa_01_qyk7lwxp, but exist several json files and five fasta files. I can successfully assemble the paired reads into a complete contig, and the command I used is: $ abyss-pe np=8 *name=/data/results/atram_out_220407/kaM_49223_abyss* k=64 in='results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_seq.fa_01_qyk7lwxp/paired_1.fasta results/atram_out_220407/kaM_49223_imd/atram_ke04m6v2/kaM_49223_seq.fa_01_qyk7lwxp/paired_2.fasta' I found that *when we set the output file name in abyss, the path must be absolute*. Maybe you had noticed it when writing aTRAM? Thanks. — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AZ3QXTRJKYEO7P5YMXYTVSFCTNANCNFSM5SY6AWBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>