Add support for Chinese and Japanese stop words #507
Conversation
Signed-off-by: Sarah Yurick <[email protected]>
| stop_lists: A dictionary stop lists, where the keys are languages (e.g., "ENGLISH") | ||
| and the values are Python frozensets denoting the list of stop words for that language. | ||
| If None, it defaults to jusText's stop lists: https://github.com/miso-belica/jusText/tree/main/justext/stoplists, | ||
| with added Thai, Chinese, and Japanese support. |
There was a problem hiding this comment.
While I think it's important for NeMo Curator to support Thai, Chinese, and Japanese, I also think it would be a good idea for us to allow users to pass in their own stop lists as a workaround.
This way, if a language is not already supported, the user can do it themselves. Additionally, a user might not like the stop lists provided by jusText and want to pass in their own custom stop lists for that reason, too.
| @@ -128,6 +128,7 @@ def extract_text(self, html, stop_words): | |||
| paragraphs = handler.paragraphs | |||
|
|
|||
| # Context free classification | |||
| # TODO: Check Thai, Chinese, Japanese, and Korean | |||
There was a problem hiding this comment.
Words in Thai, Chinese, Japanese, and Korean are not separated by spaces. I need to make sure to either (1) raise a warning about the stop word logic used by our jusText/Resiliparse extractors (such as suggesting to modify the stopwords_low/stopwords_high/required_stopword_density parameters?) or (2) adding word splitting logic like I did for #320.
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Closes #459.
Related PRs:
get_word_splitter#320