Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] refactor of dataset builder and executor #537

Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open

[WIP] refactor of dataset builder and executor #537

wants to merge 40 commits into from

Conversation

cyruszhang
Copy link
Collaborator

Key elements of this PR:

  1. YAML显式定义dataset不同来源；local和remote分开定义
  2. 更加灵活开放的数据集参数化控制；根据不同来源，支持不同参数和相关验证；并留出口子支持更多追加/细节配置
  3. 解绑Executor的hardcode支持（目前RayExecutor只接受local json格式，并在代码层面hardcode绑定）；Executor/RayExecutor不绑定dataset输入格式，但是根据formatter/downloader对于executor类型的支持来判断是否可加载
  4. 提高Executor框架扩展性，以更方便支持Nemo、Dask、Spark等其他引擎
  5. 支持数据格式验证
  6. 额外的数据来源支持
    a. 支持modelscope
    b. 支持arxiv，下载、解压、引入
    c. 支持wiki，下载、解压、引入
    d. 支持commoncrawl，下载、解压、引入
  7. 兼容命令行目前的dataset_path格式
  8. 兼容数据混搭，data mixture
  9. 兼容empty_formatter/generated_dataset_config通路

design doc: https://aliyuque.antfin.com/yilei.z/cnk4dn/qomvqql62lyglrh2?singleDoc# 《Dataset/Loader/Executor的重构设计》

@cyruszhang
ignore __dj__produced_data__
d11f89c
@cyruszhang
add download framework; add wiki support
41dea26
@cyruszhang
refactor formatter; add dataset_builder
50f8d3d
@cyruszhang
merge with master
817caab
@cyruszhang
add config files and test entry
a089de4
@cyruszhang
initial dataset_builder
5a717d7
@cyruszhang
Merge branch 'main' into feat/cyruszhang/data-downloader
9c79844
@cyruszhang
add mixture dataset support; type/subtype
ffba7e7
@cyruszhang
RayExecutor with ExecutorBase
79ae980
@cyruszhang
get rid of subtype for local dataset; depending on ext for proper rou…
e6a6e71 
…ting
@cyruszhang
use source instead of sub_type for remote dataset configs
eb300f0
@cyruszhang
arxiv downloader return Dataset instead of DJDataset
456eea1
@cyruszhang
rewrite CLI datapath with test cases
c25e40f
@cyruszhang
add executor and dataload strategy logic
75ffe3f
@cyruszhang
Merge branch 'main' into feat/cyruszhang/data-downloader
4ec1ef9
@cyruszhang
add layered load strategies
4fb6e17
@cyruszhang
Merge branch 'main' into feat/cyruszhang/data-downloader
84803cd
@cyruszhang
fix circular dependency; add dataset config test
cb5b80a
@cyruszhang
update dataset_path parsing in config
daf7a85
@cyruszhang
fix download test case; add wildcard matching for load strategy
7c48892
@cyruszhang
add test case for load strategy wild card matching
940b44d
@cyruszhang
add more test cases for datapath rewrite logic; fix rewrite to handle…
b80f991 
… space in file name
@cyruszhang
materialize symlinks for duplicates
0d5d4ba
@cyruszhang
add load strategy validation framework
f3a4ec4
@cyruszhang
add DataValidator logic
70fffd2
@cyruszhang
data validator as separate pre-processing
bbc303d
@cyruszhang
update data validator logic and add/fix test cases
4b6065f
@cyruszhang
[nit] rename test
0b153ab
@cyruszhang
[nit] rename test again
171b361
@cyruszhang
add builder test cases; update ds config validation logic
6841d19
@cyruszhang cyruszhang requested review from HYLcool and pan-x-c January 10, 2025 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@drcege drcege Awaiting requested review from drcege

@BeachWang BeachWang Awaiting requested review from BeachWang

@yxdyc yxdyc Awaiting requested review from yxdyc

@HYLcool HYLcool Awaiting requested review from HYLcool

@pan-x-c pan-x-c Awaiting requested review from pan-x-c

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@cyruszhang