Add support for Chinese and Japanese stop words #507
Conversation
Signed-off-by: Sarah Yurick <[email protected]>
nemo_curator/download/commoncrawl.py
Outdated
| stop_lists: A dictionary stop lists, where the keys are languages (e.g., "ENGLISH") | ||
| and the values are Python frozensets denoting the list of stop words for that language. | ||
| If None, it defaults to jusText's stop lists: https://github.com/miso-belica/jusText/tree/main/justext/stoplists, | ||
| with added Thai, Chinese, and Japanese support. |
There was a problem hiding this comment.
While I think it's important for NeMo Curator to support Thai, Chinese, and Japanese, I also think it would be a good idea for us to allow users to pass in their own stop lists as a workaround.
This way, if a language is not already supported, the user can do it themselves. Additionally, a user might not like the stop lists provided by jusText and want to pass in their own custom stop lists for that reason, too.
nemo_curator/download/commoncrawl.py
Outdated
| @@ -128,6 +128,7 @@ def extract_text(self, html, stop_words): | |||
| paragraphs = handler.paragraphs | |||
|
|
|||
| # Context free classification | |||
| # TODO: Check Thai, Chinese, Japanese, and Korean | |||
There was a problem hiding this comment.
Words in Thai, Chinese, Japanese, and Korean are not separated by spaces. I need to make sure to either (1) raise a warning about the stop word logic used by our jusText/Resiliparse extractors (such as suggesting to modify the stopwords_low/stopwords_high/required_stopword_density parameters?) or (2) adding word splitting logic like I did for #320.
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
nemo_curator/download/commoncrawl.py
Outdated
|
|
||
| stop_list_dict["THAI"] = thai_stopwords | ||
| if lang_key in ["THAI", "CHINESE", "JAPANESE"]: |
There was a problem hiding this comment.
Nit: Make this conditional if lang_key in custom_stopwords
nemo_curator/download/commoncrawl.py
Outdated
| @@ -152,7 +158,19 @@ def extract_text(self, html, stop_words): | |||
| self.max_heading_distance, | |||
| ) | |||
|
|
|||
| return [p.text for p in paragraphs if not p.is_boilerplate] | |||
| if self.is_boilerplate is None: | |||
| if language in ["THAI", "CHINESE", "JAPANESE", "KOREAN"]: | |||
There was a problem hiding this comment.
can you abstract away this list of 4 languages to a global var that is shared between all of the extractors?
There was a problem hiding this comment.
Updated, thanks!
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
docs/user-guide/download.rst
Outdated
| 1. Decode the HTML within the record from binary to text. | ||
| 2. If the HTML can be properly decoded, then with `pyCLD2 <https://github.com/aboSamoor/pycld2>`_, perform language detection on the input HTML. | ||
| 3. Finally, the extract the relevant text with `jusText <https://github.com/miso-belica/jusText>`_ or `Resiliparse <https://github.com/chatnoir-eu/chatnoir-resiliparse>`_ from the HTML and write it out as a single string within the 'text' field of a json entry within a `.jsonl` file. | ||
| 1. Decode the HTML within the record from binary to text. |
There was a problem hiding this comment.
I just took a peak at the rendered version of this page and it looks a little messed up. Can you fix it?
There was a problem hiding this comment.
Updated, let me know what you think.
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
| if language in NON_SPACED_LANGUAGES: | ||
| warnings.warn( | ||
| "stopword_density is ignored for non-space-separated languages." | ||
| ) | ||
| result = paragraphs |
There was a problem hiding this comment.
Since #431 was just merged, I added this logic to the Trafilatura extractor. It is the same as the Resiliparse logic.
Closes #459.
Related PRs:
get_word_splitter#320