Wrapper for Arrow Datasets & Dataset Pieces #754
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a wrapper around Arrow's ParquetDataset legacy class, to allow us to re-implement that class's API using Arrow's new dataset class. Add a wrapper around Arrow's ParquetDatasetPiece legacy class, to allow us to re-implement that class's API using Arrow's new dataset "Fragment" class.
|
Dan Lidral-Porter seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Codecov Report
@@ Coverage Diff @@
## master #754 +/- ##
==========================================
+ Coverage 86.27% 86.38% +0.11%
==========================================
Files 85 86 +1
Lines 5084 5141 +57
Branches 787 793 +6
==========================================
+ Hits 4386 4441 +55
Misses 559 559
- Partials 139 141 +2
Continue to review full report at Codecov.
|
selitvin
reviewed
Apr 29, 2022
| from pyarrow import parquet as pq | ||
|
|
||
|
|
||
| class PetastormPyArrowDataset: |
There was a problem hiding this comment.
Please add a comment explaining why do we need the wrapper.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a wrapper around PyArrow's ParquetDataset and ParquetDatasetPieces, the first part of the effort to support PyArrow's new Dataset API that was discussed in issue #613.
Since this wrapper has no functionality of its own, I didn't add unit tests specifically for the wrapper; let me know if I should.
I was unable to get the tests in
petastorm/tests/test_tf_dataset.pyto complete when running them inside the provided docker container, even when I left them to run for over 24 hours. Given that this is only a wrapper, and the code paths where the wrapper is used are covered by other tests, I think it's unlikely that tests in that file would fail when all other tests pass, but I'll keep an eye on the CI results. If they fail, then don't bother reviewing this until I fix them up (which may take several rounds of pushing new commits, as I can only run those tests in CI and not locally).