[BUG] PEP 420 namespace packages via package_dir don't work with editable installs

[mgorny]

setuptools version

78.1.0

Python version

3.13.3

OS

Gentoo Linux amd64

Additional environment information

No response

Description

I'm trying to port Triton's setup.py to stop using symlinks to cross-link Python packages and use package_dir instead. Unfortunately, it seems that if a package referenced through package_dir does not have a __init__.py (i.e. is effectively a PEP 420 namespace package), it cannot be imported from an editable install.

Expected behavior

All packages being importable from an editable install.

How to Reproduce

mkdir -p python/test third_party/{a,b}
> python/test/__init__.py
> third_party/a/__init__.py
> third_party/a/foo.py
> third_party/b/foo.py
cat > setup.py <<EOF
from setuptools import setup
setup(name="test",
      packages=["test", "test.a", "test.b"],
      package_dir={"": "python",
                   "test.a": "third_party/a",
                   "test.b": "third_party/b"})
EOF
pip install -e .
python -c 'import test.a.foo; import test.b.foo'

Output

$ pip install -e .
Obtaining file:///tmp/repro
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: test
  Building editable for test (pyproject.toml) ... done
  Created wheel for test: filename=test-0.0.0-0.editable-py3-none-any.whl size=2570 sha256=ede19ca9e89c0f19de29833a8d0466abf67ab9b99dd30d04237612a997803cd0
  Stored in directory: /tmp/pip-ephem-wheel-cache-ijmh_p0o/wheels/0e/11/03/c16968b602979cd363b907ce63c485bbbc0baada53c46b3cc5
Successfully built test
Installing collected packages: test
Successfully installed test-0.0.0
$ python -c 'import test.a.foo; import test.b.foo'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import test.a.foo; import test.b.foo
                       ^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'test.b'

Note that wheel has a working file structure, by comparison:

$ unzip -l dist/test-0.0.0-py3-none-any.whl 
Archive:  dist/test-0.0.0-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  04-11-2025 16:14   test/__init__.py
        0  04-11-2025 16:14   test/a/__init__.py
        0  04-11-2025 16:14   test/a/foo.py
        0  04-11-2025 16:14   test/b/foo.py
       48  04-11-2025 16:15   test-0.0.0.dist-info/METADATA
       91  04-11-2025 16:15   test-0.0.0.dist-info/WHEEL
        5  04-11-2025 16:15   test-0.0.0.dist-info/top_level.txt
      559  04-11-2025 16:15   test-0.0.0.dist-info/RECORD
---------                     -------
      703                     8 files

[abravalheri]

Hi @mgorny , I believe that with a complex package_dir like this one, the package is being installed via a meta path finder.

The problem is that meta path finders are tricky, and the import machinery has some quirks when it comes to namespaces. I believe this is described in the docs as part of the limitations: https://setuptools.pypa.io/en/latest/userguide/development_mode.html#limitations.

So the TL;DR is that I played around with this use case before, but I did not manage to find a full blown solution given the state of the ecosystem.

Things could be a bit different different if wheels allowed symlinks.

---

Let me try to run the reproducer locally to see if I find something different than what I had investigated before

[abravalheri]

Let me try to run the reproducer locally to see if I find something different than what I had investigated before

So I run the following:

> docker run --rm -it python:3.13-bookworm /bin/bash

python3 -m venv /tmp/venv

mkdir /tmp/pkg
cd /tmp/pkg

mkdir -p python/test third_party/{a,b}
touch python/test/__init__.py
touch third_party/a/__init__.py
touch third_party/a/foo.py
touch third_party/b/foo.py

cat > setup.py <<EOF
from setuptools import setup
setup(name="test",
      packages=["test", "test.a", "test.b"],
      package_dir={"": "python",
                   "test.a": "third_party/a",
                   "test.b": "third_party/b"})
EOF

cat << EOF > pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
EOF

/tmp/venv/bin/python -m pip install -e .

As you mentioned, if I run python -c 'import test.a.foo; import test.b.foo', the command will fail.

So I opened the REPL to investigate the following:

>>> import sys
>>> editable = sys.meta_path[-1]
>>> editable.find_spec('test.b.foo')
ModuleSpec(name='test.b.foo', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fea76ead010>, origin='/tmp/pkg/third_party/b/foo.py')

So we can see here that, if it would come to the meta path finder itself, it would have found the module, but I suspect that the import machinery is not reaching that stage and it halts before reaching the meta path finder.

[abravalheri]

There is an open question, discussion in python/cpython#92054 related to this.

I believe we would need some input from the cpython team about how to proceed.

But if any member of the community finds a way to support this use case that does not depend on inputs from the cpython team and would like to contribute, it would be great.