-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for Chinese and Japanese stop words #507
Conversation
Signed-off-by: Sarah Yurick <[email protected]>
nemo_curator/download/commoncrawl.py
Outdated
stop_lists: A dictionary stop lists, where the keys are languages (e.g., "ENGLISH") | ||
and the values are Python frozensets denoting the list of stop words for that language. | ||
If None, it defaults to jusText's stop lists: https://github.com/miso-belica/jusText/tree/main/justext/stoplists, | ||
with added Thai, Chinese, and Japanese support. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I think it's important for NeMo Curator to support Thai, Chinese, and Japanese, I also think it would be a good idea for us to allow users to pass in their own stop lists as a workaround.
This way, if a language is not already supported, the user can do it themselves. Additionally, a user might not like the stop lists provided by jusText and want to pass in their own custom stop lists for that reason, too.
nemo_curator/download/commoncrawl.py
Outdated
@@ -128,6 +128,7 @@ def extract_text(self, html, stop_words): | |||
paragraphs = handler.paragraphs | |||
|
|||
# Context free classification | |||
# TODO: Check Thai, Chinese, Japanese, and Korean |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Words in Thai, Chinese, Japanese, and Korean are not separated by spaces. I need to make sure to either (1) raise a warning about the stop word logic used by our jusText/Resiliparse extractors (such as suggesting to modify the stopwords_low
/stopwords_high
/required_stopword_density
parameters?) or (2) adding word splitting logic like I did for #320.
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few minor points.
nemo_curator/download/commoncrawl.py
Outdated
|
||
stop_list_dict["THAI"] = thai_stopwords | ||
if lang_key in ["THAI", "CHINESE", "JAPANESE"]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Make this conditional if lang_key in custom_stopwords
nemo_curator/download/commoncrawl.py
Outdated
@@ -152,7 +158,19 @@ def extract_text(self, html, stop_words): | |||
self.max_heading_distance, | |||
) | |||
|
|||
return [p.text for p in paragraphs if not p.is_boilerplate] | |||
if self.is_boilerplate is None: | |||
if language in ["THAI", "CHINESE", "JAPANESE", "KOREAN"]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you abstract away this list of 4 languages to a global var that is shared between all of the extractors?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated, thanks!
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
docs/user-guide/download.rst
Outdated
1. Decode the HTML within the record from binary to text. | ||
2. If the HTML can be properly decoded, then with `pyCLD2 <https://github.com/aboSamoor/pycld2>`_, perform language detection on the input HTML. | ||
3. Finally, the extract the relevant text with `jusText <https://github.com/miso-belica/jusText>`_ or `Resiliparse <https://github.com/chatnoir-eu/chatnoir-resiliparse>`_ from the HTML and write it out as a single string within the 'text' field of a json entry within a `.jsonl` file. | ||
1. Decode the HTML within the record from binary to text. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just took a peak at the rendered version of this page and it looks a little messed up. Can you fix it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated, let me know what you think.
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
if language in NON_SPACED_LANGUAGES: | ||
warnings.warn( | ||
"stopword_density is ignored for non-space-separated languages." | ||
) | ||
result = paragraphs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since #431 was just merged, I added this logic to the Trafilatura extractor. It is the same as the Resiliparse logic.
Closes #459.
Related PRs:
get_word_splitter
#320