Skip to content
/ corpus_filter Public

Scripts for machine translation corpora filtering/ 机器翻译平行语料过滤的脚本

8 stars 2 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

alphadl/corpus_filter

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
discard
discard
 
 
save
save
 
 
README.md
README.md
 
 
cn_raw_1000
cn_raw_1000
 
 
en_raw_1000
en_raw_1000
 
 
ja.valid
ja.valid
 
 
kr.val
kr.val
 
 
parallel_corpus_clean.py
parallel_corpus_clean.py
 
 
zh.val
zh.val
 
 
zh.valid
zh.valid
 
 

Repository files navigation

该脚本有过滤语料的作用

This Script is Mainly for Filtering the Paralle Corpora

过滤特征有：(Features)

1.双语句子长度比率(length ratio of src and tgt sentences)

2.重复句子(repeated sentences)

目前支持的语种类：(Supported LANGs)

  • 英文EN<-->中文ZH
  • 日文JP<-->中文ZH
  • 韩文KR<-->中文ZH

9th/Aug/2018

About

Scripts for machine translation corpora filtering/ 机器翻译平行语料过滤的脚本

Topics

machine-translation machine-translation-data-processing machine-translation-metrics

Resources

Readme
Activity

Stars

8 stars

Watchers

2 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages