New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Projection pushdown optimization #4180
base: main
Are you sure you want to change the base?
Conversation
This reverts commit f660086.
95f199a
to
9c32232
Compare
return std::nullopt; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should I remove these braces added on auto save?
libtenzir/src/type.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can discard type.cpp changes
(void)order; | ||
return nullptr; | ||
} | ||
bool selection_optimized = false; | ||
bool expression_optimized = false; | ||
bool order_optimized = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had to add these to distinguish which type of optimization had occurred.
Here, we implement projection pushdown, allowing the select operator to get pushed up the pipeline and to allow for optimization in feather and parquet parse. In this PR, we modified every inherited optimize function to include a selection type, we allowed for feather and parquet to read only selected columns, and considered cases where select can be pushed up. This is a good starting point for the summarize and put operators because the framework for moving selection information has been implemented.
Semantically, an empty columnar selection means that no selection should be performed. Thus, special care must be taken when dealing with a null selection. Here, a null selection acts as a blocker. Further, when returning an optimize result, std::null opt, the selection is blocked. Thus, order_invariant and do_not_optimize are blockers
Currently, only slice, batch, and pass allow the selection to move upstream. More should be added carefully.