Option to disable --sanitize / override collection dependencies

[jplitza]

We have the situation that a collection we use depends on a module foo (without version). We would like to patch that version, and included a line in our requirements.txt like

foo @ git+https://github.com/...@deadbeefdeadbeefdeadbeefdeadbeef

But since ansible-builder introspect has a hardcoded --sanitize argument, our VCS specification is completely ignored and merged to only foo with the collection requirements.

[SirUli]

So that is about a python requirements.txt which has special settings which are dropped.

The problem here is likely related to #319 since the library that performs this parsing is more than 3 years unmaintained and doesn't understand many of the pip features.

[jplitza]

I saw that issue but doubt it's related. It may be a dependency to fix this issue, but the builder code explicitly acts upon the requirement's name only:

ansible-builder/ansible_builder/requirements.py

Lines 31 to 36 in 1ad38e4

	if req.name in seen_pkgs:
	for prior_req in consolidated:
	if req.name == prior_req.name:
	prior_req.specs.extend(req.specs)
	prior_req.collections.append(collection)
	break

[SirUli]

Yep, tested that now. It is partly due to that. While the logic in the ansible-builder code only compares names and appends specs then, the library requirements-parser also doesn't yield the required feedback:

lines = ['foo @ git+https://github.com/@deadbeefdeadbeefdeadbeefdeadbeef']
for req in requirements.parse('\n'.join(lines)):
    print(req.line)
    print(req.editable)
    print(req.local_file)
    print(req.specifier)
    print(req.vcs)
    print(req.revision)
    print(req.name)
    print(req.uri)
    print(req.subdirectory)
    print(req.path)
    print(req.hash_name)
    print(req.extras)
    print(req.specs)

Leads to:

foo @ git+https://github.com/@deadbeefdeadbeefdeadbeefdeadbeef
False
False
True
None
None
foo
None
None
None
None
[]
[]

The library simply cannot deal with this format. It however would understand the format as follows:

git+https://git.myproject.org/SomeProject#egg=something

But here ansible-builder doesn't move the req.uri forward. So this parsing of requirements needs a bit more attention to detail or the usage of req.line instead.

Best Regards, Uli

[jplitza]

BTW, I worked around the issue by adding

additional_build_steps:
  append:
    - RUN pip install --no-deps --force git+https://github.com/...@deadbeefdeadbeefdeadbeefdeadbeef

[Shrews]

I don't think we totally want to disable the sanitization feature (it will remove duplications), but we definitely need to update to understand some of the new pip features. An option to disable ignoring packages in the exclusion list during the sanitization process is needed, per issue #334.

[dmsimard]

I couldn't get to the bottom of the issue at this time but I was trying to include the "ansible" package in a python requirements.txt dependency and found out that ansible-builder probably "sanitized" the requirements and removed it.

It's reproduceable with a requirements.txt that's like this:

ansible
ara

Running ansible-builder build -v 3 will eventually show:

[2/3] STEP 3/4: RUN ansible-builder introspect --sanitize --user-pip=requirements.txt --write-bindep=/tmp/src/bindep.txt --write-pip=/tmp/src/requirements.txt
# Sanitized dependencies for /usr/share/ansible/collections
---
python:
- 'ara  # from collection user'
system: []

The "ansible" package should be there but it's not.

I used @jplitza's workaround and it worked (thanks!) but was surprised by the behavior.

[Shrews]

@dmsimard You may be interested in #336

[eqrx]

Hey all, putting work this on hold until #336 if through since it needs changes on the same portion of the code.