-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NDArrayMeta.__module__
makes a very expensive, unused inspect call
#110
Comments
here's a drop-in monkeypatch function for anyone else this affects def patch_npytyping():
"""
npytyping makes an expensive call to inspect.stack()
that makes imports of pydantic models take ~200x longer than
they should:
References:
- https://github.com/ramonhagenaars/nptyping/issues/110
"""
from nptyping import ndarray
from nptyping.pandas_ import dataframe
from nptyping import recarray
from nptyping import base_meta_classes
import inspect
from types import FrameType
# make a new __module__ methods for the affected classes
def new_module_ndarray(cls) -> str:
return cls._get_module(inspect.currentframe(), 'nptyping.ndarray')
def new_module_recarray(cls) -> str:
return cls._get_module(inspect.currentframe(), 'nptyping.recarray')
def new_module_dataframe(cls) -> str:
return cls._get_module(inspect.currentframe(), 'nptyping.pandas_.dataframe')
# and a new _get_module method for the parent class
def new_get_module(cls, stack: FrameType, module: str) -> str:
return "typing" if inspect.getframeinfo(stack.f_back).function == "formatannotation" else module
# now apply the patches
ndarray.NDArrayMeta.__module__ = property(new_module_ndarray)
recarray.RecArrayMeta.__module__ = property(new_module_recarray)
dataframe.DataFrameMeta.__module__ = property(new_module_dataframe)
base_meta_classes.SubscriptableMeta._get_module = new_get_module |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem
I have a module with ~several dozen Pydantic classes that have ~200
NDArray
field annotations. Importing that module takes 21.15 seconds. 19.95s (94%) are spent inNDArrayMeta.__module__
, nearly 100% of which is spent in itsinspect.stack()
call.the module call does this:
which in turn calls:
This seems to be an attempt to give a clean module name to
inspect.formatannotation
by returning "typing" which is special cased to be stripped out of the string representation used inhelp()
(see python/cpython#72176 )No matter whether importing, printing, or calling
help()
, the return of__module__
was alwaysnptyping.ndarray
, so the call appears to be entirely unused anyway.Options
Replace with more constrained
inspect
callIf instead of getting the full stack, only the current and parent frame are inspected, the problem is resolved - imports are now ~1s which is entirely pydantic's overhead -- the
__module__
call takes 0.08424s cumulative (for 591 calls).so for
NDArrayMeta
and
SubscriptableMeta
That's identical to the original
stack
call, so a 236x perf boost for free.I had other solutions i was going to test but that one worked so well i didn't bother
The text was updated successfully, but these errors were encountered: