Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default string dtype should not raise fallback performance warnings #58581

Open
Tracked by #54792
jorisvandenbossche opened this issue May 5, 2024 · 1 comment
Open
Tracked by #54792
Labels
Arrow pyarrow functionality Strings String extension data type and string data
Milestone

Comments

@jorisvandenbossche
Copy link
Member

Issue related to the new default "string" dtype (#54792)

As long as the pyarrow-backed string dtype was opt-in, so users could get the better performance, it might have made sense to warn for certain cases where it was not actually using pyarrow but was falling back to the slower object-dtype implementation.

However, when we make the pyarrow-backed dtype the default string dtype for pandas 3.0, I don't think the current warnings make sense:

>>> pd.options.future.infer_string = True
>>> s = pd.Series(["a", "b", "c"])
>>> import re
>>> s.str.contains("a", flags=re.IGNORECASE)
PerformanceWarning: Falling back on a non-pyarrow code path which may decrease performance.
0     True
1    False
2    False
dtype: bool

Of course, it's still taking a slower code path, but 1) the user cannot do anything about this, and 2) the user also didn't choose this explicitly, it just got this dtype by default, so we don't need to warn the user that their choice for the faster dtype didn't have any effect.

So I think we should remove the warning by default (maybe still give users a way to get them optionally, now we have an option for this, xref #56921), and better document in the docstring if a certain option will cause a slowdown.

Or if we keep a warning, it should at least be an actionable warning, if that is possible. For example in the specific case above, a user could pass case=False instead of flags=re.IGNORECASE for the same result and without fallback. But then 1) the warning should actually point users to that better option, and 2) I don't know if such alternative will always exist.

cc @mroeschke @phofl

@mroeschke
Copy link
Member

I'd be OK removing this warning. (And +1 to eventually remove this fallback behavior)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Arrow pyarrow functionality Strings String extension data type and string data
Projects
None yet
Development

No branches or pull requests

2 participants