-
Notifications
You must be signed in to change notification settings - Fork 281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Petastorm hangs forever in DataBricks #804
Comments
Hi @juzzmac, I had the same problem in Databricks. Is MLflow autologging on by any chance? It seems that MLflow tries to load the dataset in memory for logging purposes, which is not possible for the endless stream that Petastorm generates when Adding the following flag in the autologging call fixed it for me:
Hope this solves it! See also: mlflow/mlflow#9600 |
I've tried several different versions of the following code, all of which work when running locally but hang forever in DataBricks
(single node, 13.3 LTS ML runtime):
The text was updated successfully, but these errors were encountered: