Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

String as input in petastorm dataloaders #795

Open
freud14-tm opened this issue May 10, 2023 · 3 comments
Open

String as input in petastorm dataloaders #795

freud14-tm opened this issue May 10, 2023 · 3 comments

Comments

@freud14-tm
Copy link

In the _sanitize_pytorch_types function used in petastorm dataloaders, an exception is raised when an input is a string:

elif re.search('[SaUO]', value.dtype.str):

Is this really necessary? Maybe the default_collate function from PyTorch might not always like, although it seems to support it according to the documentation, but this prevents making other types of collate functions.

Would you consider removing the exception? Thank you.

@selitvin
Copy link
Collaborator

Sure! Would you like to propose a PR for that? Ideally, with a small unit test showing that it works.

@freud14-tm
Copy link
Author

It seems more complicated than I thought with the current code base of the library and I am not using the library right now. Still think the idea is worthwhile but feel free to close the issue if you wish. Otherwise, let it open in order to keep the idea close.

@pratikgujjar
Copy link

Wondering if there is a reason for the string type to not be supported? A column in a parquet file could be all string labels. It is currently not possible to read this parquet file as a PyTorch dataloader becuase this line in the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants