Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix save_ckpt bug #536

Merged
merged 1 commit into from
Jan 14, 2025
Merged

fix save_ckpt bug #536

merged 1 commit into from
Jan 14, 2025

Conversation

HYLcool
Copy link
Collaborator

@HYLcool HYLcool commented Jan 9, 2025

  • IndexError if the number of samples in the result dataset is less than the number of workers when saving dataset to disk

@HYLcool
* fix save_ckpt bug: error if the number of samples in the result dat…
04b5ccb 
…aset is less than the number of workers when saving dataset to disk
@HYLcool HYLcool added bug Something isn't working dj:efficiency regarding to efficiency issues and enhancements labels Jan 9, 2025
@HYLcool HYLcool requested review from BeachWang and yxdyc January 9, 2025 09:00
@HYLcool HYLcool self-assigned this Jan 9, 2025
@HYLcool HYLcool linked an issue Jan 9, 2025 that may be closed by this pull request
[Bug]: 开启checkpoint，当配置的一个pipline执行到最后一个算子时，np>left samples，在开启checkpoint会报错。 #531
Closed
3 tasks
@HYLcool HYLcool added dj:core issues/PRs about the core functions of Data-Juicer and removed dj:efficiency regarding to efficiency issues and enhancements labels Jan 9, 2025
BeachWang
BeachWang approved these changes Jan 14, 2025
View reviewed changes
Copy link
Collaborator

@BeachWang BeachWang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BeachWang BeachWang merged commit 0575193 into main Jan 14, 2025
3 checks passed
@BeachWang BeachWang deleted the fix/save_ckpt_parallel branch January 14, 2025 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BeachWang BeachWang BeachWang approved these changes

@yxdyc yxdyc Awaiting requested review from yxdyc

Assignees

@HYLcool HYLcool

Labels
bug Something isn't working dj:core issues/PRs about the core functions of Data-Juicer
Projects
None yet
Milestone
No milestone
2 participants
@HYLcool @BeachWang