Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FutureWarning: 'ParquetDataset.partitions' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. #800

Open
ton11111 opened this issue Oct 6, 2023 · 1 comment

Comments

@ton11111
Copy link

ton11111 commented Oct 6, 2023

When I call make_reader, I keep getting the following warning in each epoch. Will this be fixed in the future?

Code

  from petastorm import make_reader
  from petastorm.pytorch import DataLoader

  reader = make_reader(
      dataset_url=f"file://train.parquet",
      shuffle_rows=False
  )
  return DataLoader(reader, batch_size=128)

Warning

/opt/conda/lib/python3.9/site-packages/petastorm/py_dict_reader_worker.py:267: FutureWarning: 'ParquetDataset.partitions' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.partitioning' attribute instead.

Here is my version.
pyarrow: 13.0.0
petastorm: 0.12.1

@tingstam
Copy link

Same issue
pyarrow 8.0.0
petastorm 0.12.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants