Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jusText not work with Chinese webpage #459

Open
yangjingyi opened this issue Dec 31, 2024 · 1 comment · May be fixed by #507
Open

jusText not work with Chinese webpage #459

yangjingyi opened this issue Dec 31, 2024 · 1 comment · May be fixed by #507
Assignees
@sarahyurick
Labels
bug Something isn't working

Comments

@yangjingyi
Copy link

Hi,

I found that jusText do not work with Chinese html data. Because languages = justext.get_stoplists() cannot get stopwords for Chinese.
https://github.com/NVIDIA/NeMo-Curator/blob/main/nemo_curator/download/commoncrawl.py

Could you please add some package or codes to make it work for Chinese data?

Thanks!
@yangjingyi yangjingyi added the bug Something isn't working label Dec 31, 2024
@sarahyurick sarahyurick self-assigned this Jan 22, 2025
@sarahyurick
Copy link
Collaborator

Hi, I can look into this.

@sarahyurick sarahyurick changed the title fasText not work with Chinese webpage jusText not work with Chinese webpage Jan 31, 2025
@sarahyurick sarahyurick linked a pull request Jan 31, 2025 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sarahyurick sarahyurick

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for Chinese and Japanese stop words sarahyurick/NeMo-Curator
2 participants
@yangjingyi @sarahyurick