-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dataloader2 with FullSyncIterDataPipe throws error during initilization #1190
Comments
@ejguan Hi can you help provide some insights you have? Great thanks! |
Are you running multiple DPP at the same time? |
@ejguan I'm only running one DDP job. The DDP job is initialized by torchx. And I got these errors while running job on AWS Batch and SageMaker, where I believe all the instances are isolated and there should be no other job running. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
馃悰 Describe the bug
Hi, we found some strange during using Dataloader2. Here's some details about the issue.
traininig_args.eval_steps
training steps.IterableDataPipe
withShardingFilterIterDataPipe
Can you help provide some context on what could be the root cause and how to fix this? Thanks!
Log:
Versions
The text was updated successfully, but these errors were encountered: