Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Chinese and Japanese stop words #507

Merged
merged 16 commits into from
Mar 7, 2025

Conversation

sarahyurick
Copy link
Collaborator

Signed-off-by: Sarah Yurick <[email protected]>
Comment on lines 415 to 418
stop_lists: A dictionary stop lists, where the keys are languages (e.g., "ENGLISH")
and the values are Python frozensets denoting the list of stop words for that language.
If None, it defaults to jusText's stop lists: https://github.com/miso-belica/jusText/tree/main/justext/stoplists,
with added Thai, Chinese, and Japanese support.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I think it's important for NeMo Curator to support Thai, Chinese, and Japanese, I also think it would be a good idea for us to allow users to pass in their own stop lists as a workaround.

This way, if a language is not already supported, the user can do it themselves. Additionally, a user might not like the stop lists provided by jusText and want to pass in their own custom stop lists for that reason, too.

@@ -128,6 +128,7 @@ def extract_text(self, html, stop_words):
paragraphs = handler.paragraphs

# Context free classification
# TODO: Check Thai, Chinese, Japanese, and Korean
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Words in Thai, Chinese, Japanese, and Korean are not separated by spaces. I need to make sure to either (1) raise a warning about the stop word logic used by our jusText/Resiliparse extractors (such as suggesting to modify the stopwords_low/stopwords_high/required_stopword_density parameters?) or (2) adding word splitting logic like I did for #320.

sarahyurick and others added 8 commits January 31, 2025 13:11
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
@sarahyurick sarahyurick marked this pull request as ready for review February 19, 2025 20:47
Signed-off-by: Sarah Yurick <[email protected]>
Copy link
Collaborator

@ryantwolf ryantwolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor points.


stop_list_dict["THAI"] = thai_stopwords
if lang_key in ["THAI", "CHINESE", "JAPANESE"]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Make this conditional if lang_key in custom_stopwords

@@ -152,7 +158,19 @@ def extract_text(self, html, stop_words):
self.max_heading_distance,
)

return [p.text for p in paragraphs if not p.is_boilerplate]
if self.is_boilerplate is None:
if language in ["THAI", "CHINESE", "JAPANESE", "KOREAN"]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you abstract away this list of 4 languages to a global var that is shared between all of the extractors?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, thanks!

sarahyurick and others added 3 commits March 6, 2025 10:25
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
@sarahyurick sarahyurick requested a review from ryantwolf March 6, 2025 18:56
1. Decode the HTML within the record from binary to text.
2. If the HTML can be properly decoded, then with `pyCLD2 <https://github.com/aboSamoor/pycld2>`_, perform language detection on the input HTML.
3. Finally, the extract the relevant text with `jusText <https://github.com/miso-belica/jusText>`_ or `Resiliparse <https://github.com/chatnoir-eu/chatnoir-resiliparse>`_ from the HTML and write it out as a single string within the 'text' field of a json entry within a `.jsonl` file.
1. Decode the HTML within the record from binary to text.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just took a peak at the rendered version of this page and it looks a little messed up. Can you fix it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, let me know what you think.

sarahyurick and others added 3 commits March 6, 2025 11:13
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Comment on lines +363 to +367
if language in NON_SPACED_LANGUAGES:
warnings.warn(
"stopword_density is ignored for non-space-separated languages."
)
result = paragraphs
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since #431 was just merged, I added this logic to the Trafilatura extractor. It is the same as the Resiliparse logic.

@sarahyurick sarahyurick requested a review from ryantwolf March 6, 2025 19:41
@sarahyurick sarahyurick merged commit 9a2bd42 into NVIDIA:main Mar 7, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

jusText not work with Chinese webpage
2 participants