I had problems in the past with compile (specially when profiling the performance), so I would prefer not to change this one.

The documentation also mentions:

Warning It is possible to crash the Python interpreter with a sufficiently large/complex string when compiling to an AST object due to stack depth limitations in Python’s AST compiler.

which might be problematic?

Closing this for now, please feel free to reopen if we have evidence that this method:

1. Does not affect the performance of the code
2. Does not increase the risk of crashes (as mentioned in the docs for compile)

@abravalheri yup, i think that's ok for now. i've been looking into the way exec/compile work in python, and will ask on the python mailinglist for input form devs.

Two additional thoughts though:

1. Thoughts on having this behind a flag?

2. There are a couple of other places in the code which do use the compile-then-exec pattern (as opposed to exec-ing source string). Should these be changed? ( my initial feeling goes along the lines of "if it ain't broke, don't fix it" )

Respectfully, I fail to see why calling exec would avoid any of the caveats of compile, since it must obviously also compile the source to bytecode before execution.

Meanwhile, the broken traceback is a real usability problem. If you are not very familiar with setuptools, you will have no clue where the error could have come from (notice that the name of bad_package does not appear at all in the traceback)

Respectfully, I fail to see why calling exec would avoid any of the caveats of compile, since it must obviously also compile the source to bytecode before execution.

Hi @mcejp, I understand why they in principle should behave the same, however this is not what I have observed in the past via profiling.

Could you please submit a profiling study showing that the times are similar?
In the past I have used https://gist.github.com/abravalheri/d561ca67405a9b3b1d182811a0eaf398#file-readme-md

When using the compile I could see the following:

Without the compile the result was:

(relevant part emphasized in pink)

(These results are a couple of years old, and in Python 3.8 which is not EoL so things may have changed)

If we can obtain a better result and/or a reasonable explanation of why the profiling method is biased accompanied by a reviewed version, I would be happy to review it.

I fail to see why calling exec would avoid any of the caveats of compile

The documentation of compile explicitly says it can crash the interpreter, while exec does not mention that fact. Could you please get confirmation (e.g. from code devs and/or documentation team) that both methods are subject to the same problems (of crashing the interpreter)? I would be happy to review this under new evidence.

@abravalheri yup, i think that's ok for now. i've been looking into the way exec/compile work in python, and will ask on the python mailinglist for input form devs.

Two additional thoughts though:

1. Thoughts on having this behind a flag?
2. There are a couple of other places in the code which do use the compile-then-exec pattern (as opposed to exec-ing source string). Should these be changed? ( my initial feeling goes along the lines of "if it ain't broke, don't fix it" )

I apologize @MarcinKonowalczyk for missing this comment.

Regarding feature flags, I am a little reluctant to add more levers and knobs to setuptools as it is very complicated. We however recently introduced setuptools.compat.py310.add_note so maybe one thing we can do with help debugging is to add a note to the exception mentioning setup.py (we would have to think a little bit more about it but that is a first idea).

Regarding the compile-then-exec pattern, I aggree that better not to change. But it would be interesting to know where they are used, I suspect that only in deprecated code paths or when setuptools is doing some kind of introspection on the byte-code.