Wrong quay image requested when attemping test locally due to different format version.py generated by versioneer

[DrYak]

Cross posting this issue from bioconda-recipes.

I posted it initially over-there as it was a result of following the "contributing to bioconda" procedure, but given that this is a bug when invoking bioconda-utils build, it might actually be more relevant to post about it here.

[DrYak]

I suspect that this line of code is the culprit given that every call to this constructor leaves the parameter to its default (except in the tests).

It should be better to filter the __version__ string to only the subpart that corresponds to the parts used for quay.io image tags.

[dpryan79]

How did you install bioconda-utils? This should normally only happen if you install it from source rather than via conda.

[DrYak]

On this last attempt I've run the bootstrap.pyexecutable. (While doing the previous 0.3.5 release of smallgenomeutilities, I remember having similarish issue, both after attempting to update the build environment (conda update --all) and a doing a clean re-install with bootstrap).

So the bioconda-utils package is definitely not self-compiled, but fetched from bioconda.

[DrYak]

Interestingly, install the package myself, doesn't reproduce this:

$ mamba create -n biocu-test bioconda-utils
…
$ conda activate biocu-test
$ ls ${CONDA_PREFIX}/lib/python*/site-packages/bioconda_utils/_version.py
/home/dryak/miniconda/envs/biocu-test/lib/python3.7/site-packages/bioconda_utils/_version.py
$ grep 'version' ${CONDA_PREFIX}/lib/python*/site-packages/bioconda_utils/_version.py
…
 "version": "0.17.10"

[DrYak]

I would still suggest using a different default strategy:

docker_base_image='quay.io/bioconda/bioconda-utils-build-env-cos7:{}'.format(__version__.split("+")[0])

As, I presume, such longer tags aren't expected anyway on quay.io

[DrYak]

The steps used yesterday according to my history are:

$ ./bootstrap.py /tmp/bioconda-build
$ source ~/.config/bioconda/activate
$ bioconda-utils lint --git-range master
$ bioconda-utils build --docker --mulled-test --git-range master

[DrYak]

Trying again this morning triggers the exact same error:

$ conda deactivate
$ rm -rf /tmp/bioconda-build/
$ ./bootstrap.py /tmp/bioconda-build
$ source ~/.config/bioconda/activate
$ grep 'version' ${CONDA_PREFIX}/lib/python*/site-packages/bioconda_utils/_version.py
…
 "version": "0.17.10+1.g3bf462e"
…
$ bioconda-utils build --force --docker --mulled-test --packages cojac/0.2
…
10:24:22 BIOCONDA INFO BUILD START recipes/cojac/0.2
10:24:22 BIOCONDA INFO (COMMAND) docker run -t --net host --rm -v /tmp/tmp7rhz6u0l/build_script.bash:/opt/build_script.bash -v /tmp/bioconda-build/miniconda/conda-bld/:/opt/host-conda-bld -v /home/dryak/project/bioconda-recipes/recipes/cojac/0.2:/opt/recipe -e LANG=en_US.UTF-8 -e LC_CTYPE=en_US.UTF-8 -e HOST_USER_ID=1000 quay.io/bioconda/bioconda-utils-build-env-cos7:0.17.10_1.g3bf462e /bin/bash /opt/build_script.bash
10:24:22 BIOCONDA INFO (ERR) Unable to find image 'quay.io/bioconda/bioconda-utils-build-env-cos7:0.17.10_1.g3bf462e' locally
10:24:24 BIOCONDA INFO (ERR) docker: Error response from daemon: manifest for quay.io/bioconda/bioconda-utils-build-env-cos7:0.17.10_1.g3bf462e not found: manifest unknown: manifest unknown.
10:24:24 BIOCONDA INFO (ERR) See 'docker run --help'.
10:24:24 BIOCONDA ERROR COMMAND FAILED (exited with 125): docker run -t --net host --rm -v /tmp/tmp7rhz6u0l/build_script.bash:/opt/build_script.bash -v /tmp/bioconda-build/miniconda/conda-bld/:/opt/host-conda-bld -v /home/dryak/project/bioconda-recipes/recipes/cojac/0.2:/opt/recipe -e LANG=en_US.UTF-8 -e LC_CTYPE=en_US.UTF-8 -e HOST_USER_ID=1000 quay.io/bioconda/bioconda-utils-build-env-cos7:0.17.10_1.g3bf462e /bin/bash /opt/build_script.bash

10:24:24 BIOCONDA ERROR BUILD FAILED recipes/cojac/0.2
10:24:24 BIOCONDA INFO (COMMAND) conda build purge
10:24:25 BIOCONDA ERROR BUILD SUMMARY: of 1 recipes, 1 failed and 0 were skipped. Details of recipes and environments follow.
10:24:25 BIOCONDA ERROR BUILD SUMMARY: FAILED recipe recipes/cojac/0.2

[DrYak]

So to summarize:

• installing just the bioconda-utils package with mamba: works as intended.
• installing the whole environment with bootstrap.py: fails, because a differently tagged package arrives and quay.io doesn't have a matching docker image.
  • this problem doesn't seem to be transient

[dpryan79]

Excellent, we need to deprecate the boostrap.py and clarify this in the documentation. The version number code stuff is actually intentional, since I think we're using it to test the resulting container as part of the CI testing for bioconda-utils, but I'll look into this more.

@daler I know you were hoping to update the documentation, so see above for some ideas for areas needing improvement.

[DrYak]

Re: documentation improvement:

The best would be to edit the documentation and recommend people to do something like:

# create separate environment dedicated to test building packages:
mamba create -n bioconda-utils -c conda-forge -c bioconda bioconda-utils

conda activate bioconda-utils

bioconda-utils lint --git-range master
bioconda-utils build --docker --mulled-test --git-range master

(PS: I'm not 100% sure my command-line channels' options is the correct order, but it seems to prioritize conda-forge, so I hope it's correct?)

[dpryan79]

The channel order is correct. We want to prioritize conda-forge so dependencies built there take priority over older versions that used to be supplied via bioconda :)