python_requires fails when remote is in CONAN_REMOTES but not added to conan #331

madebr · 2019-01-31T20:26:54Z

This happens on CI.

When using a python_requires in a recipe that is available on a remote but not in the local cache,
the ConanMultiPackager constructor throws because while loading the conanfile class, the remotes in CONAN_REMOTES is not yet added to the remotes of conan.

reproducer:

YOURUSERNAME=... #madebr
YOURREMOTEURL=... #https://api.bintray.com/conan/madebr/maarten
YOURPASSWORD=... #0123456789abcdef01234567890

mkdir /tmp/cpt_test
cd /tmp/cpt_test
mkdir pkg_base && cd pkg_base
cat >conanfile.py <<EOF
from conans import ConanFile
class PkgBase(ConanFile):
    name = "pkg_base"
    version = "1.2"
EOF

cd /tmp/cpt_test
cat >conanfile.py  <<EOF
from conans import python_requires

base = python_requires("pkg_base/1.2@testremote/testing")

class Pkg(base.PkgBase):
    name = "pkg"
EOF

cat >build.py <<EOF
from conan.packager import ConanMultiPackager
builder = ConanMultiPackager()
builder.add_common_builds()
builder.run()
EOF

conan remote add -f testremote $YOURREMOTEURL
conan export pkg_base pkg_base/1.2@testremote/testing
conan user -r testremote -p "$YOURPASSWORD" "$YOURUSERNAME"
conan upload pkg_base/1.2@testremote/testing -r testremote
conan remote remove testremote
conan remove pkg_base -f

CONAN_PASSWORD=$YOURPASSWORD CONAN_USERNAME=$YOURUSERNAME CONAN_REMOTES=$YOURREMOTEURL@True@testremote python3 build.py

The output of this last command is:

$ CONAN_PASSWORD=$YOURPASSWORD CONAN_USERNAME=$YOURUSERNAME CONAN_REMOTES=$YOURREMOTEURL@True@testremote python3 build.py

****************************************************************************************************

   ____ ____ _____    __   ____                          ____            _                      _____           _      __  
  / ___|  _ \_   _|  / /  / ___|___  _ __   __ _ _ __   |  _ \ __ _  ___| | ____ _  __ _  ___  |_   _|__   ___ | |___  \ \ 
 | |   | |_) || |   | |  | |   / _ \| '_ \ / _` | '_ \  | |_) / _` |/ __| |/ / _` |/ _` |/ _ \   | |/ _ \ / _ \| / __|  | |
 | |___|  __/ | |   | |  | |__| (_) | | | | (_| | | | | |  __/ (_| | (__|   < (_| | (_| |  __/   | | (_) | (_) | \__ \  | |
  \____|_|    |_|   | |   \____\___/|_| |_|\__,_|_| |_| |_|   \__,_|\___|_|\_\__,_|\__, |\___|   |_|\___/ \___/|_|___/  | |
                     \_\                                                           |___/                               /_/ 

Version: 0.23.0fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

 >> Branch detected
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
pkg_base/1.2@testremote/testing: Not found in local cache, looking in remotes...
pkg_base/1.2@testremote/testing: Trying with 'conan-center'...
pkg_base/1.2@testremote/testing: Trying with 'bincrafters'...
Traceback (most recent call last):
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/loader.py", line 45, in load_class
    return self.cached_conanfiles[conanfile_path]
KeyError: '/tmp/cpt_test/conanfile.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/python_requires.py", line 33, in _look_for_require
    python_require = self._cached_requires[require]
KeyError: 'pkg_base/1.2@testremote/testing'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/loader.py", line 254, in _parse_conanfile
    loaded = imp.load_source(module_id, conan_file_path)
  File "/usr/lib64/python3.6/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/cpt_test/conanfile.py", line 3, in <module>
    base = python_requires("pkg_base/1.2@testremote/testing")
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/python_requires.py", line 58, in __call__
    python_req = self._look_for_require(require)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/python_requires.py", line 41, in _look_for_require
    recorder=ActionRecorder())
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/proxy.py", line 34, in get_recipe
    result = self._get_recipe(ref, check_updates, update, remote_name, recorder)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/proxy.py", line 50, in _get_recipe
    remote, new_ref = self._download_recipe(ref, output, remote_name, recorder)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/proxy.py", line 157, in _download_recipe
    raise NotFoundException(msg)
conans.errors.NotFoundException: Unable to find 'pkg_base/1.2@testremote/testing' in remotes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build.py", line 2, in <module>
    builder = ConanMultiPackager()
  File "/home/maarten/.local/lib/python3.6/site-packages/cpt/packager.py", line 177, in __init__
    conanfile = load_cf_class(os.path.join(self.cwd, "conanfile.py"), self.conan_api)
  File "/home/maarten/.local/lib/python3.6/site-packages/cpt/packager.py", line 32, in load_cf_class
    return conan_api._loader.load_class(path)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/loader.py", line 48, in load_class
    _, conanfile = parse_conanfile(conanfile_path, self._python_requires)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/loader.py", line 231, in parse_conanfile
    module, filename = _parse_conanfile(conanfile_path)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/loader.py", line 275, in _parse_conanfile
    '\n'.join(trace[3:])))
conans.errors.ConanException: Unable to load conanfile in /tmp/cpt_test/conanfile.py
KeyError: '/tmp/cpt_test/conanfile.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/python_requires.py", line 33, in _look_for_require
    python_require = self._cached_requires[require]
KeyError: 'pkg_base/1.2@testremote/testing'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/loader.py", line 254, in _parse_conanfile
    loaded = imp.load_source(module_id, conan_file_path)
  File "/usr/lib64/python3.6/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/cpt_test/conanfile.py", line 3, in <module>
    base = python_requires("pkg_base/1.2@testremote/testing")
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/python_requires.py", line 58, in __call__
    python_req = self._look_for_require(require)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/python_requires.py", line 41, in _look_for_require
    recorder=ActionRecorder())
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/proxy.py", line 34, in get_recipe
    result = self._get_recipe(ref, check_updates, update, remote_name, recorder)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/proxy.py", line 50, in _get_recipe
    remote, new_ref = self._download_recipe(ref, output, remote_name, recorder)
  File "/home/maarten/.local/lib/python3.6/site-packages/conans/client/graph/proxy.py", line 157, in _download_recipe
    raise NotFoundException(msg)
conans.errors.NotFoundException: Unable to find 'pkg_base/1.2@testremote/testing' in remotes

The text was updated successfully, but these errors were encountered:

uilianries · 2019-01-31T20:29:57Z

python_requires + CPT? This is an evil bug! 👿

lasote · 2019-07-01T10:33:21Z

As a workaround, you can pass the reference attribute to the ConanMultiPackager or use the CONAN_REFERENCE to specify at least lib/version so you can avoid the early load of the conanfile. Let me know if it helps.

madebr · 2019-07-01T16:20:18Z

Running

CONAN_REFERENCE=testremote/testing CONAN_PASSWORD=$YOURPASSWORD CONAN_USERNAME=$YOURUSERNAME CONAN_REMOTES=$YOURREMOTEURL@True@testremote python3 build.py

gives the following output:

****************************************************************************************************

   ____ ____ _____    __   ____                          ____            _                      _____           _      __  
  / ___|  _ \_   _|  / /  / ___|___  _ __   __ _ _ __   |  _ \ __ _  ___| | ____ _  __ _  ___  |_   _|__   ___ | |___  \ \ 
 | |   | |_) || |   | |  | |   / _ \| '_ \ / _` | '_ \  | |_) / _` |/ __| |/ / _` |/ _` |/ _ \   | |/ _ \ / _ \| / __|  | |
 | |___|  __/ | |   | |  | |__| (_) | | | | (_| | | | | |  __/ (_| | (__|   < (_| | (_| |  __/   | | (_) | (_) | \__ \  | |
  \____|_|    |_|   | |   \____\___/|_| |_|\__,_|_| |_| |_|   \__,_|\___|_|\_\__,_|\__, |\___|   |_|\___/ \___/|_|___/  | |
                     \_\                                                           |___/                               /_/ 

Version: 0.28.1fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

 >> Branch detected
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

[local_vars]
+---------------------------+-----------------------+
| Configuration             | value                 |
|---------------------------+-----------------------|
| cwd                       | /tmp/ss               |
| username                  | madebr                |
| skip_check_credentials    | False                 |
| upload_only_when_stable   | False                 |
| upload_only_when_tag      | False                 |
| upload_only_recipe        | False                 |
| specified_channel         | testing               |
| stable_channel            | stable                |
| channel                   | testing               |
| partial_reference         | testremote/testing    |
| conanfile                 | /tmp/pkg_conanfile.py |
| use_docker                | False                 |
| docker_conan_home         | /home/conan           |
| sudo_docker_command       | sudo -E               |
| sudo_pip_command          |                       |
| pip_command               | pip                   |
| docker_shell              | /bin/sh -c            |
| docker_platform_param     |                       |
| lcow_user_workaround      |                       |
| exclude_vcvars_precommand | False                 |
| upload_dependencies       |                       |
| docker_32_images          | False                 |
| force_selinux             | False                 |
| conan_pip_package         | conan==1.16.0         |
| vs10_x86_64_enabled       | False                 |
| builds_in_current_page    | []                    |
+---------------------------+-----------------------+
pkg_base/1.2@testremote/testing: Not found in local cache, looking in remotes...
pkg_base/1.2@testremote/testing: Trying with 'conan-center'...
pkg_base/1.2@testremote/testing: Trying with 'bincrafters'...
pkg_base/1.2@testremote/testing: Trying with 'conan-transit'...
Traceback (most recent call last):
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/graph/python_requires.py", line 44, in _look_for_require
    python_require = self._cached_requires[require]
KeyError: 'pkg_base/1.2@testremote/testing'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/graph/python_requires.py", line 75, in __call__
    python_req = self._look_for_require(require)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/graph/python_requires.py", line 53, in _look_for_require
    recorder=ActionRecorder())
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/graph/proxy.py", line 33, in get_recipe
    result = self._get_recipe(ref, check_updates, update, remotes, recorder)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/graph/proxy.py", line 49, in _get_recipe
    remote, new_ref = self._download_recipe(ref, output, remotes, remotes.selected, recorder)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/graph/proxy.py", line 164, in _download_recipe
    raise NotFoundException(msg)
conans.errors.NotFoundException: Unable to find 'pkg_base/1.2@testremote/testing' in remotes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/loader.py", line 53, in load_class
    _, conanfile = parse_conanfile(conanfile_path, self._python_requires)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/loader.py", line 253, in parse_conanfile
    module, filename = _parse_conanfile(conanfile_path)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/loader.py", line 287, in _parse_conanfile
    loaded = imp.load_source(module_id, conan_file_path)
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/pkg_conanfile.py", line 3, in <module>
    base = python_requires("pkg_base/1.2@testremote/testing")
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/graph/python_requires.py", line 79, in __call__
    raise ConanException('Unable to find python_requires("{}") in remotes'.format(require))
conans.errors.ConanException: Unable to find python_requires("pkg_base/1.2@testremote/testing") in remotes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pkg_build.py", line 3, in <module>
    builder.add_common_builds()
  File "/home/maarten/.local/lib/python3.7/site-packages/cpt/packager.py", line 422, in add_common_builds
    conanfile = load_cf_class(os.path.join(self.cwd, self.conanfile), self.conan_api)
  File "/home/maarten/.local/lib/python3.7/site-packages/cpt/packager.py", line 46, in load_cf_class
    return conan_api._loader.load_class(path)
  File "/home/maarten/.local/lib/python3.7/site-packages/conans/client/loader.py", line 59, in load_class
    raise ConanException("Error loading conanfile at '{}': {}".format(conanfile_path, e))
conans.errors.ConanException: Error loading conanfile at '/tmp/pkg_conanfile.py': Unable to find python_requires("pkg_base/1.2@testremote/testing") in remotes

It proceeds further but does not look like to use the reference in the CONAN_REMOTES environment variable.

lasote · 2019-07-02T06:49:42Z

I need to investigate that. It looks like a different issue, maybe with Conan.

lasote · 2019-07-02T08:04:52Z

Yeah, you would need to use also the shared_option_name argument in the add_common_builds as part of the workaround. Obviously, this should be improved. I'll try

lasote · 2019-07-02T08:39:44Z

This PR has been merged: #359
I think that it should solve the issue even with docker usage. It will be released in the next release. Thanks

madebr mentioned this issue Jan 31, 2019

[RFC] Two recipes for regular flex and flex_installer in one repo: use python_requires bincrafters/conan-flex#5

Closed

This was referenced Mar 21, 2019

delaying load of recipe until it is required by builders #358

Open

add remotes in constructor #359

Merged

lasote added the type: bug label Jul 2, 2019

lasote closed this as completed in #359 Jul 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python_requires fails when remote is in CONAN_REMOTES but not added to conan #331

python_requires fails when remote is in CONAN_REMOTES but not added to conan #331

madebr commented Jan 31, 2019

uilianries commented Jan 31, 2019

lasote commented Jul 1, 2019

madebr commented Jul 1, 2019

lasote commented Jul 2, 2019

lasote commented Jul 2, 2019

lasote commented Jul 2, 2019

python_requires fails when remote is in CONAN_REMOTES but not added to conan #331

python_requires fails when remote is in CONAN_REMOTES but not added to conan #331

Comments

madebr commented Jan 31, 2019

uilianries commented Jan 31, 2019

lasote commented Jul 1, 2019

madebr commented Jul 1, 2019

lasote commented Jul 2, 2019

lasote commented Jul 2, 2019

lasote commented Jul 2, 2019