Issue when trying to run test

[mrmrwinter]

Hi,

I am looking to run Longstitch on my assembly, and need ntlinks installed to do so. I have installed from source code, but when running the test script i receive the following error:

+ ntLink scaffold -B target=scaffolds_1.fa reads=long_reads_1.fa w=250
indexlr --long --pos --strand -k 32 -w 250 -t 4 scaffolds_1.fa > scaffolds_1.fa.k32.w250.tsv
sh -c 'pigz -p4 -f -cd long_reads_1.fa | \
indexlr --long --pos --strand -k 32 -w 250 -t 4 - | \
/home/mike/software/ntLink-1.3.4/bin/ntlink_pair.py -p scaffolds_1.fa.k32.w250.z1000 -n 1 -m scaffolds_1.fa.k32.w250.tsv -s scaffolds_1.fa  \
-k 32 -a 1 -z 1000 -f 10 -x 0  --verbose -'
Running pairing stage of ntLink ...

Parameters:
	Reads TSV files:  ['-']
	-s  scaffolds_1.fa
	-m  scaffolds_1.fa.k32.w250.tsv
	-p  scaffolds_1.fa.k32.w250.z1000
	-n  1
	-k  32
	-a  1
	-z  1000
	-f  10
	-x  0.0
2022-08-03 10:20:20.910853 : Reading minimizers scaffolds_1.fa.k32.w250.tsv
2022-08-03 10:20:20.911557 : Reading fasta file scaffolds_1.fa
2022-08-03 10:20:20.911647 : Finding pairs
2022-08-03 10:20:20.922134 : Building scaffold graph
2022-08-03 10:20:20.922208 : Filtering the graph
2022-08-03 10:20:20.922255 : Printing graph scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot
2022-08-03 10:20:20.922371 : DONE!
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 1 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n1.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n1.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 2 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n2.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n2.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 3 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n3.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n3.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 4 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n4.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n4.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 5 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n5.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n5.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 6 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n6.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n6.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 7 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n7.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n7.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 8 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n8.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n8.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 9 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n9.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n9.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 10 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n10.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n10.abyss-scaffold.path.sterr'
/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py --min_n 1 --max_n 10  -p out \
-g scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot --conservative scaffolds_1.fa.k32.w250.z1000.n1.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n2.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n3.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n4.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n5.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n6.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n7.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n8.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n9.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n10.abyss-scaffold.path -o scaffolds_1.fa.k32.w250.z1000.stitch.path
Running ntLink stitch paths stage...

2022-08-03 10:20:21.083186  : Finding optimal n...
Traceback (most recent call last):
  File "/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py", line 486, in <module>
    main()
  File "/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py", line 482, in main
    NtLinkPath().main()
  File "/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py", line 426, in main
    best_file = self.find_optimal_n(self.args.PATH)
  File "/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py", line 387, in find_optimal_n
    best_n = int(name_match.group(1))
AttributeError: 'NoneType' object has no attribute 'group'
make: *** [/home/mike/software/ntLink-1.3.4/ntLink:203: scaffolds_1.fa.k32.w250.z1000.stitch.path] Error 1

I can't find any mention of this error in relation to ntlinks on Google, so any help would be appreciated.

Many thanks

Edit: i am also getting the same error when trying to run longstitch on my own data

[warrenlr]

hello Michael, how did you install longstitch? If you haven't done so, I'd recommend installing longstitch+dependencies using conda & make a new environment https://anaconda.org/bioconda/longstitch

thanks for your interest in ntlink/longstitch

conda install -c bioconda longstitch

[mrmrwinter]

Hi,

I initially tried to conda install in a fully fresh environment but kept getting an error saying it required python 3.1, whereas the conda install installed 3.10. I tried force installing 3.1 into the environment but it wouldn't resolve. I replicated this error on two machines.

I then tried the brew install method, which failed with a missing ntlinks_bottle_manifest, leading me to try the source code install.

The source install seemingly works, and runs both longstitch and ntlinks tests up until the error I mentioned above

[mrmrwinter]

Accidentally closed, sorry.

[mrmrwinter]

Ive tried again to do the conda install
I ran:
conda create -n longstitch2
conda activate longstitch2
conda install -c bioconda longstitch

But it hung for hours solving the environment:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment:

i then tried installing it using mamba:
'mamba install -c bioconda longstitch`

Which returned:

Looking for: ['longstitch']

bioconda/noarch                                             Using cache
pkgs/main/noarch                                   814.7kB @   2.4MB/s  0.3s
pkgs/r/noarch                                        1.3MB @   2.0MB/s  0.3s
pkgs/r/linux-64                                      1.4MB @   1.4MB/s  0.3s
pkgs/main/linux-64                                   4.6MB @   4.0MB/s  1.2s
bioconda/linux-64                                    4.2MB @   3.5MB/s  1.2s
conda-forge/noarch                                   8.9MB @   4.0MB/s  2.3s
conda-forge/linux-64                                24.5MB @   3.9MB/s  6.4s

Pinned packages:
  - python 3.10.*

Encountered problems while solving:
  - package longstitch-1.0.1-hdfd78af_0 requires ntlink, but none of the providers can be installed

I then tried installing ntlink from conda:
conda install -c bioconda ntlink

Which returned:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.

ResolvePackageNotFound: 
  - python=3.1

I then tried with mamba which returned this:

Looking for: ['ntlink']

bioconda/linux-64                                           Using cache
bioconda/noarch                                             Using cache
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
pkgs/r/linux-64                                               No change
pkgs/main/noarch                                              No change
pkgs/r/noarch                                                 No change
pkgs/main/linux-64                                            No change

Pinned packages:
  - python 3.10.*

Encountered problems while solving:
  - package ntlink-1.0.1-py39h39abbe0_0 requires python >=3.9,<3.10.0a0, but none of the providers can be installed

I then force installed python 3.9:
conda install python=3.9 -f

This worked, and after this i could succesfully install both ntlink and longstitch using mamba.

I ran longstitch on my assembly and it ran to completion.

[warrenlr]

Thank you for documenting your install attempts.

I am glad to hear of the successful install+execution of longstitch on your system.

For posterity (&other users), would you be able to also record the spec of your system on here?

[mrmrwinter]

Sure.

I'm running on Ubuntu 20.4
Miniconda version 4.10.3

CPU:Ryzen9 3900X
GPU: GTX1660 Super
RAM 64Gb

[lcoombe]

Thanks for the info @mrmrwinter, and I'm glad you got it working!

To confirm, dependencies of LongStitch do require python 3.8 or 3.9 (as you found when installing ntLink separately), which I believe is leading to the initial error you saw with mamba install -c bioconda longstitch. conda install mamba installs python 3.10 by default in tests on my end.

Just to reiterate for any future users who have the issue, if using mamba, either use:

• conda install mamba python=3.9 prior to mamba install -c longstitch
• mamba install -c bioconda python=3.9 longstitch

[Adamtaranto]

Hi @lcoombe, I'm running into a similar issue.

Do you know what parts of ntLink are actually incompatible with Python >= 3.10? Can they easily be upgraded?

[lcoombe]

Hi @Adamtaranto,

Thanks for bringing up this issue again. At least on the ntLink side, that restriction was added by a bioconda maintainer in 2022 - there shouldn't be any ntLink code that is incompatible.
I'll do some tests on my end, and see if updating the ntLink conda recipe itself will do the trick, or if there are other dependencies that cause issues. Will keep you updated!

[Adamtaranto]

I can't see anywhere in the bioconda recipe or in setup.py that is specifically enforcing python <= 3.10. Maybe coming from a dependency?

I also noticed that ntLink can't be installed from bioconda on an M-series mac as there is no osx-arm64 build of the package. You might be able to make the package conda installable on Apple silicon macs by including "noarch: python" in the build section.

I see that the bioconda recipe requires " - {{ compiler('cxx') }}", but it's not clear to me if anything is getting compiled. So might be fine as noarch?

[lcoombe]

Hi @Adamtaranto,

The python3 requirement is in the meta.yaml for the recipe: https://github.com/bioconda/bioconda-recipes/blob/86d1f45fa2731bb8a3fbdb6ef157b38a04254885/recipes/ntlink/meta.yaml#L13
I could see that the line in question was changed by a maintainer using git blame.

Thanks for pointing out the compiler part - you're right, previous versions of ntLink did require compilation, but not since we refactored to make btllib a dependency (vs. packaged in ntLink). I have a busy week, but will aim to open a bioconda PR to update the ntLink recipe this week - I will let you know when I do.

[lcoombe]

I have opened a PR on bioconda (bioconda/bioconda-recipes#45522) to update the recipe to noarch (thus removing the python version restriction). Hopefully it will be merged soon!
Testing the build locally, I found the updates to work fine with python 3.10

[lcoombe]

@Adamtaranto - the update to the ntLink conda recipe has now been merged. Let me know if you are still seeing any issues!

[Adamtaranto]

Hi @lcoombe - Nearly there! No error from ntlink itself, but still fails to install on M-series mac as Abyss has no osx-arm64 build.
I think you can force Bioconda packages to make an Apple Silicon compatible build by including this script in the recipe: bioconda/bioconda-recipes#23454 (comment)

The Python <= 3.10 limitation seems to be coming from btllib, though I can't see where that is being enforced.

Thanks for looking into this.

[lcoombe]

Hi @Adamtaranto,

Thanks for your patience with all this!

Looking into it more, the issue is really on the bioconda side. They are working on rolling out python 3.11 and ARM builds - but the last update from them is that it is still a work in progress: bioconda/bioconda-recipes#33333 (comment)
Unfortunately, as far as I can tell, the script that you linked does not look like something that belongs in a recipe - it looks like code that the maintainers are developing to integrate ARM support in the building of artifacts during their CI? (ex. we never call bioconda-utils build in a build.sh script).

At this point, unfortunately for packages that cannot be noarch, bioconda doesn't have support for builds on python 3.11 or osx-arm64. For now, your options would be to use python 3.10 and a different machine, or build the tools from source. It looks like these improvements are just on the horizon on the bioconda side, though, so I will keep an eye on the developments.

[lcoombe]

Hi @Adamtaranto - In case it helps you in the meantime - I saw this issue on bioconda while keeping an eye on the repository for ARM build updates: bioconda/bioconda-recipes#45586
They describe a potential workaround for installing conda packages on ARM architectures - I haven't tried it myself, but could be helpful.

[lcoombe]

Just to update in case anyone is still interested - bioconda recently began to support builds on the osx-arm64 architecture.
The relevant ntLink dependencies now have those builds available (abyss and btllib), so installing ntLink on M-series macs should now work 👍

[Adamtaranto]

Yay! That's awesome! Have you found any good examples/tutorials for how to add osx-arm64 builds to existing recipes?

[lcoombe]

Yeah, so it's opt-in right now, and a pretty easy addition to the meta.yaml file for the recipes - described here: bioconda/bioconda-recipes#33333 (comment)

Of course, the more complicated part comes when dependencies of the recipe don't yet have the appropriate builds, or if certain recipes fail the builds for whatever reason on those architectures - but hopefully over the coming months more of those problem recipes will be sorted out!