We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
starts_with
@adriangb implemented PruningPredicate support for prefix matching LIKE / NOT LIKE in
PruningPredicate
LIKE
NOT LIKE
like
However, it isn't currently supported for the starts_with function
I would like predicate pruning to happen for the starts_with function as well
So queries like
select * from my_file where starts_with(col, 'http://')
Could also use starts_with to prune parquet files
The challenge at the moment is that PruningPredicate can't refer directly to the function implementations
Given how optimized LIKE is one possible solution would be to change starts_with so it didn't just call an arrow kernel, but instead was rewritten
https://github.com/apache/datafusion/blob/main/datafusion/functions/src/string/starts_with.rs
So for example, it could be rewritten into Expr::Like by implementing simplity:
Expr::Like
simplity
https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.simplify
We could do something similar with ends_with as well
ends_with
No response
The text was updated successfully, but these errors were encountered:
I think this is a good first issue as rewriting a function should be straightforward and doesn't require indepth knowledge of the rest of the engine
Sorry, something went wrong.
take
I think the rewrite would look something like
Rewritten to the equivalent of
select * from my_file where col LIKE 'http://%'
simplify
jatin510
Successfully merging a pull request may close this issue.
Is your feature request related to a problem or challenge?
@adriangb implemented
PruningPredicate
support for prefix matchingLIKE
/NOT LIKE
inlike
expressions (prefix matching) #12978However, it isn't currently supported for the
starts_with
functionDescribe the solution you'd like
I would like predicate pruning to happen for the
starts_with
function as wellSo queries like
Could also use starts_with to prune parquet files
Describe alternatives you've considered
The challenge at the moment is that
PruningPredicate
can't refer directly to the function implementationsGiven how optimized LIKE is one possible solution would be to change
starts_with
so it didn't just call an arrow kernel, but instead was rewrittenhttps://github.com/apache/datafusion/blob/main/datafusion/functions/src/string/starts_with.rs
So for example, it could be rewritten into
Expr::Like
by implementingsimplity
:https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.simplify
We could do something similar with
ends_with
as wellAdditional context
No response
The text was updated successfully, but these errors were encountered: