How to use Data-Juicer to process Chinese documents

[aruig666]

Before Asking 在提问之前

• I have read the README carefully. 我已经仔细阅读了 README 上的操作指引。

• I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking 先搜索，再提问

• I have searched the Data-Juicer issues and found no similar questions. 我已经在 issue列表 中搜索但是没有发现类似的问题。

Question

I collected some Chinese documents and converted them into josn1 format, trying to process them through data-juicer.

I used the default configuration in /configs/demo/process.yaml, and only changed the input and output file addresses.

# Process config example for dataset

# global parameters
project_name: 'demo-process'
dataset_path: '/root/autodl-tmp/MDHB/json'  # path to your dataset directory or file
np: 4  # number of subprocess to process your dataset

export_path: '/root/autodl-tmp/MDHB/data_juicer/result.jsonl'

# process schedule
# a list of several process operators with their arguments
process:
  - language_id_score_filter:
      lang: 'zh'
      min_score: 0.8

The error report is as follows:

Could you help me see how I should modify the configuration to use data-juicer?
Thanks!

test.txt

Additional 额外信息

No response

[HYLcool]

Hi @aruig666 , thanks for your attention and using on Data-Juicer!

According to the information you provide, it seems that Data-Juicer could not find any valid files in the input directory /root/autodl-tmp/MDHB/json. I guess it's caused by the file extensions, which should be ".jsonl" (means json lines) instead of ".json1". The last letter should be lower case of letter 'L' instead of digit one '1'.

Please check and retry. If you have any further questions, contact us anytime~

[night666e]

嗨，感谢您的关注和对 Data-Juicer 的使用！

根据您提供的信息，Data-Juicer 似乎在输入目录中找不到任何有效文件。我猜这是由文件扩展名引起的，它应该是“.jsonl”（表示 json 行）而不是“.json1”。最后一个字母应该是字母 'L' 的小写，而不是数字 1 的 '1'。/root/autodl-tmp/MDHB/json

请检查并重试。如果您有任何其他问题，请随时与我们联系~

RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing.
在执行官方命令:python tools/process_data.py --config configs/demo/process.yaml

[aruig666]

Thanks, I get it! @HYLcool @night666e