Autoprofile is not working as expected.

[Erotemic]

@ta946 I'm running into an issue when attempting to use autoprofiling.

Maybe I'm remember how to use it wrong, but if I am, there is something unintuitive about the design we came up with that we might want to address in either the docs or in code changes.

The goal is that I'd like to autoprofile an entire package when running a command line tool. Unfortunately kernprof can't run a python module (e.g. python -m modname), it has to run a script (e.g. python modname.py), which makes this annoying. Ideally if I wanted to profile pip, I could do something like: python -m kernprof -l --prof-mod pip -m pip, but that's a separate issue and we can work around it by writing a script that calls the module main function of interest:

echo "
from pip._internal.cli.main import main as _main
_main()
" > my_script_v1.py
chmod +x my_script_v1.py


python -m kernprof -l \
    --prof-mod pip \
    ./my_script_v1.py freeze

python -m line_profiler -rmt "my_script_v1.py.lprof"

Now, I would expect that because we are asking it to profile a package directory that all functions and imports in pip itself would be decorated, but that doesn't seem to be the case. They are recognized, but there seems to be a rule that filters them out.

However, if I explicitly import some of the modules beforehand, I will end up profiling their contents:

echo "
from pip._internal.cli.main import main as _main
import pip._vendor
import pip._internal.utils.hashes
_main()
" > my_script_v2.py
chmod +x my_script_v2.py


python -m kernprof -l \
    --prof-mod pip \
    ./my_script_v2.py freeze

python -m line_profiler -rmt "my_script_v2.py.lprof"

Thinking about this: maybe the problem is that the modules aren't loaded when the AST transformer does its decorations? We probably don't want to do anything that would force a different import order than would happen when running the program naturally, but maybe there is a way to hook into the import system and decorate any module that is within the requested package and imported after profiling starts?

[Erotemic]

Also noting that when it does autoprofile it seems to miss setter and getter properties

[ta946]

ok just looked into whats happening. this was more of a design decision we made previously..

When you add pip to prof_mod, it finds pip._vendor and pip._internal.utils.hashes, as well as 350+ other submodule files, so autoprofile is aware of them. But to profile them, you would need to have them imported so that line profiler can decorate it

But since they are hidden from the main script, we chose to not profile them since it would mean trying to profile everything recursively (which might require importing those subomdules into the top of the AST then wrapping in profile(submodule)), so we decided you would have to explicitly enter which submodules you want profiled

Not sure what the best approach is here..
in an ideal world i would follow the _main() submodule and find the imports in there, but that would require opening the file and repeating the same process we do for the main script, and it might need to be done recursively for every import

As for the setter & getter, never needed to profile so never tested it.. would be good to have a test case we can play around with

[TTsangSC]

Hi both, on the topic of recursive auto-profiling of packages, and at the risk of looking like a shameless plug...

... in a tool¹ that I've recently been working on, I wrote some meta-path hooks so that arbitrary modules can be rewritten at import time, regardless of whether they are directly imported by the (test) script at hand. This seems to solve the issue at hand, but this solution appears to me (1) a bit intrusive and (2) potentially beyond the scope of this project.

But of course you would be much better judges for whether the above concerns are valid, or if I was over-thinking it. If you think that aspects of the tool do fall in the scope of line_profiler and should be worked back into it,² I'll be glad to make it work together.

Cheers!

P.S. Since you mentioned LineProfiler.__call__() missing handling for properties, maybe we can also add that to #328? Here's a sample implementation that I'm using in said tool.

Footnotes

1. https://gitlab.com/TTsangSC/pytest-autoprofile ↩

2. Several of my recent issues (and their proposed solutions) and PRs (ENH: auto-profile module execution #323, BUG: LineProfiler.__call__() bugged on class and static methods #328, Auto-profiling misses async defs #330) are directly inspired by my working on said tool. ↩