Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement column projection #1443

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Implement column projection #1443

wants to merge 11 commits into from
+193 −5

Conversation

gabeiglio
Copy link

@gabeiglio gabeiglio commented Dec 18, 2024 •
edited by Fokko
Loading

This is a fix for issue #1401. In which table scans needed to infer partition column by following the column projection rules

Fixes #1401

@Fokko Fokko self-requested a review December 18, 2024 20:38
@gabeiglio gabeiglio marked this pull request as ready for review December 19, 2024 15:12
kevinjqliu
kevinjqliu reviewed Dec 19, 2024
View reviewed changes
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few comments, please take a look! The PR looks great already. Thanks for working on this!

pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
Fokko
Fokko reviewed Dec 20, 2024
View reviewed changes
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
Fokko
Fokko reviewed Dec 20, 2024
View reviewed changes
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
@kevinjqliu kevinjqliu self-requested a review December 23, 2024 19:04
Removed file_schema check and initial-default logic, separated projec…
fee24ab 
…tion logic to helper method, changed test to use high-level table scan
Fokko
Fokko reviewed Dec 29, 2024
View reviewed changes
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
Fokko
Fokko reviewed Dec 29, 2024
View reviewed changes
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
Fokko
Fokko reviewed Dec 29, 2024
View reviewed changes
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
Fokko
Fokko reviewed Dec 29, 2024
View reviewed changes
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
kevinjqliu
kevinjqliu reviewed Jan 3, 2025
View reviewed changes
tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
kevinjqliu
kevinjqliu reviewed Jan 7, 2025
View reviewed changes
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
@Fokko Fokko self-requested a review January 13, 2025 12:48
Fokko
Fokko reviewed Jan 13, 2025
View reviewed changes
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
kevinjqliu
kevinjqliu reviewed Jan 20, 2025
View reviewed changes
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally LGTM! I added a few nit comments and some clarifying questions on testing.

thanks for working on this!

tests/io/test_pyarrow.py Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
kevinjqliu
kevinjqliu reviewed Feb 1, 2025
View reviewed changes
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like CI caught an interesting case where a new identity partition is added after data files were written. The accessor then cannot find the proper partition record... We need to do something like this

tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Show resolved Hide resolved
Fokko
Fokko reviewed Feb 4, 2025
View reviewed changes
Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
kevinjqliu
kevinjqliu approved these changes Feb 5, 2025
View reviewed changes
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @gabeiglio !!

@kevinjqliu kevinjqliu requested a review from Fokko February 5, 2025 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kevinjqliu kevinjqliu kevinjqliu approved these changes

@Fokko Fokko Awaiting requested review from Fokko

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

API table.scan does not conform to Iceberg spec for identity partition columns
3 participants
@gabeiglio @Fokko @kevinjqliu