Skip to content

Setuptools should not rely on pkg_resources #3085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jaraco opened this issue Feb 5, 2022 · 11 comments · May be fixed by #5007
Open

Setuptools should not rely on pkg_resources #3085

jaraco opened this issue Feb 5, 2022 · 11 comments · May be fixed by #5007

Comments

@jaraco
Copy link
Member

jaraco commented Feb 5, 2022

This issue basically tracks the root goal of obviating pkg_resources, in particular that Setuptools depends on it.

Over the past few years, myself and others have worked hard to create importlib.resources and importlib.metadata in the goal of superseding pkg_resources. These libraries are now very stable and replacing much of the functionality provided by pkg_resources.

In python/importlib_metadata#38, I examined the setuptools package to determine its level of dependency on pkg_resources. Unsurprisingly, there is a diverse set of interfaces on which setuptools relies.

Nevertheless, this project should work toward Setuptools adopting the importlib features in place of pkg_resources.

@romuald
Copy link

romuald commented Feb 14, 2022

This somehow breaks projects entry points to pyramid/paste it seems, which aren't able to find their entry point anymore (I have no idea what's the cause)

Error for Google:

LookupError: Entry point 'main' not found in egg 'my.package' (dir: xxx/.pyenv-python3.7/lib/python3.7/site-packages; protocols: paste.app_factory, paste.composite_factory, paste.composit_factory; entry_points: )

@armanckeser
Copy link

The same pr causes this issue I think #3102

@jaraco
Copy link
Member Author

jaraco commented May 20, 2025

Now that the setuptools code base no longer depends on pkg_resources, I'm considering once again republishing pkg_resources as a separate package.

I've reactivated the pypa/pkg_resources repo and re-run a filter operation on this repo to re-capture the history of that code there:

git filter-repo --path pkg_resources/ --path api_tests.txt --path pkg_resources.py --path pkg_resources.txt --path setuptools/tests/test_resources.py tests/api_tests.txt tests/test_pkg_resources.py

I'm now pondering publishing that code to see if the setuptools tests can continue to run by requiring that package at test time.

@jaraco
Copy link
Member Author

jaraco commented May 21, 2025

Attempting to remove pkg_resources, the pbr tests fail with this traceback:

  File "/Users/jaraco/draft/.venv/lib/python3.13/site-packages/pbr/core.py", line 53, in <module>
    from pbr import util
  File "/Users/jaraco/draft/.venv/lib/python3.13/site-packages/pbr/util.py", line 75, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

It seems that pbr is reliant on the interface.

@jaraco
Copy link
Member Author

jaraco commented May 21, 2025

pbr uses pkg_resources in exactly one place, to evaluate markers. That's definitely a deprecated use and superseded by 'importlib.metadata' and 'packaging'.

@jaraco
Copy link
Member Author

jaraco commented May 21, 2025

I tried uploading a copy of pkg_resources to PyPI, but it seems that name is blocked:

 pkg_resources main 🐚 twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading pkg_resources-1.0.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.0/45.0 kB • 00:00 • 346.9 MB/s
WARNING  Error during upload. Retry with the --verbose option for more details.                                                                          
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/                                                                                   
         The user 'jaraco' isn't allowed to upload to project 'pkg-resources'. See https://pypi.org/help/#project-name for more information.             

I don't think it's worth trying to reclaim that name, so I'm going to assume that supplying a separate pkg_resources package to supplement setuptools without pkg_resources. Environments that rely on setuptools will simply have to install setuptools<81.

@jaraco jaraco linked a pull request May 21, 2025 that will close this issue
2 tasks
@abravalheri
Copy link
Contributor

I don't think it's worth trying to reclaim that name, so I'm going to assume that supplying a separate pkg_resources package to supplement setuptools without pkg_resources. Environments that rely on setuptools will simply have to install setuptools<81.

It is a bit unfortunate that we cannot publish to PyPI. I can see a couple of situations in which it would be useful. If we want to be very conservative and try to reduce backwards incompatibility we could:

  • We could return pkg_resources in setuptools.build_meta:__legacy__.get_requires_for_build_sdist/wheel (either by default, or by using an heuristic like searching for the pkg_resources string in setup.py.

@cdce8p
Copy link
Contributor

cdce8p commented May 27, 2025

I tried uploading a copy of pkg_resources to PyPI, but it seems that name is blocked:

...

I don't think it's worth trying to reclaim that name, so I'm going to assume that supplying a separate pkg_resources package to supplement setuptools without pkg_resources. Environments that rely on setuptools will simply have to install setuptools<81.

Does the project need to be named pkg_resources? Wouldn't something like pkg_resources_backport (or similar) work?

You're right that replacing the deprecated APIs isn't an issue, at least not for actively maintained projects. What I'm a bit concerned about is unnecessarily breaking those which haven't received an update in a while. Especially in larger projects those dependencies aren't always avoidable.

For Home Assistant we had a similar situation with the move to Python 3.13. The removal of stdlib modules broke a lot of packages. We wouldn't be able to use 3.13 now if it weren't for backport packages. I'd assume it will be similar for the pkg_resources removal. Yes, pinning setuptools is an option but not a very good one IMO.

@jaraco
Copy link
Member Author

jaraco commented May 27, 2025

To be sure, if there's a common/impactful scenario where the presence of a backfill project would help, I'll definitely work on creating that project. The problem is that the scenarios I've thought of where it might help would be just as easily addressed by pinning Setuptools. After all, if you can require or install pkg_resources, that's the point where you could also require setuptools<81, no? The dependency is at runtime and not build time, so the complications that come with being a build time dependency are moot, I think.

If users are noticing the warning now, let's explore those cases and see what options are available.

@cdce8p
Copy link
Contributor

cdce8p commented May 27, 2025

After all, if you can require or install pkg_resources, that's the point where you could also require setuptools<81, no?

Yes, though I'd prefer to not have to do that as it "forever" limits which setuptools version is used. There might be other packages in the future which depend on a newer version and would thus be incompatible.

"forever" might be an exaggeration but some packages might not have received an update in over 3 years. Those are unlikely to get fixed. So the setuptools pin would have to stay until these are eventually removed.

The dependency is at runtime and not build time, so the complications that come with being a build time dependency are moot, I think.

If it were a build time dependency, I don't think there is an option to remove it. Thankfully this shouldn't be the case (?). Yes PIP_CONSTRAINTS exists but it's not easy to use for most people and until overwriting build requirements gets easier this would be the only option.

@jaraco
Copy link
Member Author

jaraco commented May 27, 2025

Yes, though I'd prefer to not have to do that as it "forever" limits which setuptools version is used. There might be other packages in the future which depend on a newer version and would thus be incompatible.

Acknowledged. And if there's a scenario that warrants this bridge, we'll deal with it then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants