Dependency cleanup

[ctcjab]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

pyproject.toml declares runtime dependencies on packages that are usually not runtime dependencies (e.g. packages you import from or shell out at runtime to executables they provide), which bloats and complicates the dependency graph and environment for consumers. (For example, build and virtualenv.) Typically these are instead more suitably declared as build or dev dependencies. Are all the runtime dependencies declared in pyproject.toml actually runtime dependencies? Note this affects the downstream conda package as well, whose meta.yaml has the same issue.

Additional Context

No response

[maresb]

Thanks for raising this issue! I'm glad to dump my current thoughts on this.

@ctcjab, are you running into any particular conflicts? If so, that would be good to know, and we can tackle those individually. But for now I'll focus on the general situation.

The current reason for this situation is as follows:

conda-lock/pyproject.toml

Lines 59 to 74 in 21ae67a

	# The following dependencies were added in the process of vendoring Poetry 2.0.1.
	"build >=1.2.1,<2.0.0",
	"cachecontrol[filecache] >=0.14.0,<0.15.0",
	"crashtest >=0.4.1,<0.5.0",
	"dulwich >=0.22.6,<0.23.0",
	"fastjsonschema >=2.18.0,<3.0.0",
	'importlib-metadata >= 4.4 ; python_version <"3.10"',
	"installer >=0.7.0,<0.8.0",
	"keyring >=25.1.0,<26.0.0",
	"pkginfo >=1.12,<2.0",
	"pyproject-hooks >=1.0.0,<2.0.0",
	"requests-toolbelt >=1.0.0,<2.0.0",
	"shellingham >=1.5,<2.0",
	"trove-classifiers >= 2022.5.19",
	"virtualenv >=20.26.6,<21.0.0",
	'xattr >=1.0.0,<2.0.0 ; sys_platform == "darwin"',

Poetry's attitude towards dependencies has been a pain point. They are firmly in the camp of "we are an application, not a library" and unfortunately since we vendor Poetry, conda-lock inherits this. In the past I have relaxed Poetry's upper bounds without consequence, but that does carry a minor risk of allowing untested versions. I have also successfully removed some dependencies without any impact on the functioning of conda-lock.

A similar complaint has also been raised by @bollwyvl in #615 (comment) and his subsequent two comments. I like his suggestion of modularity, and I currently believe it's the right thing to do, but it will take a lot of work.

In summary, some not-necessarily-exclusive approaches are:

1. Frequently revendor Poetry so that the dependencies stay hopefully reasonably up-to-date.
2. Relax upper bounds to reduce version conflicts. (Makes vendoring more difficult and risks breaking the tight coupling expected by Poetry.)
3. Prune unneeded dependencies. (Risks introducing breakage in edge cases, and makes revendoring difficult.)
4. Make conda-lock modular so that these requirements are unnecessary for pure-conda environments. (A lot of work.)
5. Interface with pyrattler (pixi) to eventually remove Poetry. (A lot of work and time.)

Any thoughts, or suggestions for additional approaches?