The TUFS Asian Language Parallel Corpus (TALPCo) is an open parallel corpus consisting of Japanese sentences and their translations into Korean, Burmese (Myanmar; the official language of the Republic of the Union of Myanmar), Malay (the national language of Malaysia, Singapore and Brunei), Indonesian, Thai, Vietnamese and English. TALPCo is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. See the paper below for the details of TALPCo.
- (For Korean, Burmese, Malay, Indonesian and English translations)
Nomoto, Hiroki, Kenji Okano, David Moeljadi and Hideo Sawada. 2018. TUFS Asian Language Parallel Corpus (TALPCo). Proceedings of the Twenty-Fourth Annual Meeting of the Association for Natural Language Processing, 436-439. - (For Thai and Vietnamese translations, and interpersonal meaning annotations)
Nomoto, Hiroki, Kenji Okano, Sunisa Wittayapanyanon and Junta Nomura. 2019. Interpersonal meaning annotation for Asian language corpora: The case of TUFS Asian Language Parallel Corpus (TALPCo). Proceedings of the Twenty-Fifth Annual Meeting of the Association for Natural Language Processing, 846-849. Supplement - (For Javanese translations)
Lestari, Sri Buti and Yoshimi Miyake. 2022. Jawago ni mirareru taishouhyougen: Shootomuubii karano youreibuseki [Second person expressions in Javanese: An analysis of examples from short movies]. Indonesia: Gengo to Bunka 28: 65-84. - (For Malay and Indonesian constituency tree annotations)
Nomoto, Hiroki. 2022. Kyokushoushugi ni motoduku heiretsu tsuriibanku no kouchiku [Building a parallel treebank based on minimalism]. Proceedings of the Twenty-Eighth Annual Meeting of the Association for Natural Language Processing, 103-107.
data_jpn.txt
Japanese (raw sentences)data_jpn-sound.txt
Japanese (links to sound files)data_jpn-token.txt
Japanese (tokenized sentences)data_jpn-IPSpkr.csv
Japanese (interpersonal meaning annotation, speaker)data_jpn-IPAddr.csv
Japanese (interpersonal meaning annotation, addressee)data_jpn-IPLex.csv
Japanese (interpersonal meaning annotation, lexical)data_jpn-prosub.jsonl
Japanese (pronoun substitute annotation)data_jpn-prosub.txt
Japanese (pronoun substitute annotation)
data_kor.txt
Korean (raw sentences)data_kor-sound.txt
Korean (links to sound files)data_kor-token.txt
Korean (tokenized sentences)data_kor-prosub.jsonl
Korean (pronoun substitute annotation)data_kor-prosub.txt
Korean (pronoun substitute annotation)
data_zsm.txt
Malay (raw sentences)data_zsm-sound.txt
Malay (links to sound files)data_zsm-token.txt
Malay (tokenized sentences)data_zsm-MWE.txt
Malay (multiword expression list)data_zsm-tree.txt
Malay (constituency tree annotation)data_zsm.jpn-zsm
Malay (partial Japanese-Malay alignment)data_zsm-IPSpkr.csv
Malay (interpersonal meaning annotation, speaker)data_zsm-IPAddr.csv
Malay (interpersonal meaning annotation, addressee)data_zsm-IPLex.csv
Malay (interpersonal meaning annotation, lexical)data_zsm-prosub.jsonl
Malay (pronoun substitute annotation)data_zsm-prosub.txt
Malay (pronoun substitute annotation)
data_ind.txt
Indonesian (raw sentences)data_ind-sound.txt
Indonesian (links to sound files)data_ind-token.txt
Indonesian (tokenized sentences)data_ind-MWE.txt
Indonesian (multiword expression list)data_ind-tree.txt
Indonesian (constituency tree annotation)data_ind.jpn-ind
Indonesian (partial Japanese-Indonesian alignment)data_ind-IPSpkr.csv
Indonesian (interpersonal meaning annotation, speaker)data_ind-IPAddr.csv
Indonesian (interpersonal meaning annotation, addressee)data_ind-IPLex.csv
Indonesian (interpersonal meaning annotation, lexical)data_ind-prosub.jsonl
Indonesian (pronoun substitute annotation)data_ind-prosub.txt
Indonesian (pronoun substitute annotation)
data_jav.txt
Javanese (raw sentences)data_jav-prosub.jsonl
Javanese (pronoun substitute annotation)data_jav-prosub.txt
Javanese (pronoun substitute annotation)
data_tha.txt
Thai (raw sentences)data_tha-sound.txt
Thai (links to sound files)data_tha-token.txt
Thai (tokenized sentences)data_tha.jpn-tha
Thai (partial Japanese-Thai alignment)data_tha-IPSpkr.csv
Thai (interpersonal meaning annotation, speaker)data_tha-IPAddr.csv
Thai (interpersonal meaning annotation, addressee)data_tha-IPLex.csv
Thai (interpersonal meaning annotation, lexical)data_tha-prosub.jsonl
Thai (pronoun substitute annotation)data_tha-prosub.txt
Thai (pronoun substitute annotation)
data_vie.txt
Vietnamese (raw sentences)data_vie-sound.txt
Vietnamese (links to sound files)data_vie-token.txt
Vietnamese (tokenized sentences)data_vie-MWE.txt
Vietnamese (multi-syllable expression list)data_vie.jpn-vie
Vietnamese (partial Japanese-Vietnamese alignment)data_vie-IPSpkr.csv
Vietnamese (interpersonal meaning annotation, speaker)data_vie-IPAddr.csv
Vietnamese (interpersonal meaning annotation, addressee)data_vie-IPLex.csv
Vietnamese (interpersonal meaning annotation, lexical)data_vie-prosub.jsonl
Vietnamese (pronoun substitute annotation)data_vie-prosub.txt
Vietnamese (pronoun substitute annotation)
data_myn.txt
Burmese (raw sentences)data_myn-sound.txt
Burmese (links to sound files)data_myn-token.txt
Burmese (tokenized sentences)data_myn-ps.txt
Burmese (POS-tagged sentences)data_myn-prosub.jsonl
Burmese (pronoun substitute annotation)data_myn-prosub.txt
Burmese (pronoun substitute annotation)
data_eng.txt
English (raw sentences)data_eng-US.txt
English (US) (raw sentences) [by courtesy of Charles Kelly]
readme.me
(this document)
All files are encoded in UTF-8 with DOS format.
Sentence_ID [TAB] Sentence
1176 田中さんは 学生では ありません。
1176 Mr Tanaka is not a student.
Sentence_ID [TAB] URL
Sentence_ID [LINEBREAK] token [LINEBREAK] token [LINEBREAK] <EOS>
3627
Buku
ini
mempunyai
se-
ratus
dua
puluh
muka surat
.
<EOS>
Sentence_ID [TAB] Sentence
-
White space: Phrasal boundary
-
Dash: Morpheme boundary
1176 n-pr-postp n pref-v-suf-suf
Sentence_ID.n [TAB] bracketing
(for the n
-th sentence for Sentence_ID
)
3695.2 [S [CP [Conj Tapi] [CP [C *C_decl*] [TP [DP_a [D saya]] [T'[T *T*] [AP [AP [AdvP [Adv agak]] [AP [DP *t*<a>] [A letih]]] [AdvP [Adv sedikit]]]]]]] [PU .]]
Sentence_ID [TAB] Japanese_token_index-target_language_token_index
1176 0-1 1-0 3-3 8-2 9-4
See the second paper above and its supplement for the details of the interpersonal meaning feature system.
Sentence_ID, Gender, Marital status, Honour, Age, Social status, Role, Group, Formality, Number
3243,female,,,,neutral,,,,sg
Token_index, token, Gender, Marital status, Honour, Age, Social status, Role, Group, Formality, Number
3845,,,,,,,,,,
0,Cô,female,,,elder.parents_younger_sibling,,parents_sibling.paternal,,,
1,tôi,,,,,neutral,,,,sg
2,làm việc,,,,,,,,,
3,ở,,,,,,,,,
4,cửa hàng,,,,,,,,,
5,hoa,,,,,,,,,
6,.,,,,,,,,,
<EOS>,,,,,,,,,,
The sound files for the following sentences come from TUFS Open Language Resources.
- Japanese: All sentences except sentences 1176, 1178, 1180, 1194, 1222, 1229, 1233, 1244, 1245, 1246, 1247, 1248, 1249, 1250, 1251, 1254, 1255, 1257, 1258, 1259, 1264, 1268, 1274, 1275, 1276, 1280, 1281, 1286, 1291, 1296, 1297, 1299, 1302, 1305, 1308, 1311, 1312, 1319, 1320, 1322, 1323, 1326, 1330, 1333, 1334, 1335, 1341, 1348, 1349, 1350, 1356, 1367, 1369, 1371, 1378, 1379, 1380, 1382, 1386, 1388, 1389, 1392, 1393, 1396, 1397, 1398, 1401, 1402, 1403, 1408, 1409, 1410, 1414, 1418, 1419, 1423, 1426, 1427, 1428, 1431, 1433, 1434, 1435, 1446, 1449, 1450, 1451, 1454, 1455, 1457, 1459, 1460, 1473, 1474, 1476, 1477, 1478, 1481, 1482, 1484, 1485, 1491, 1492, 1494, 1495, 1501, 1507, 1512, 1513, 1515, 1520, 1527, 1528, 1530, 1531, 1532, 1533, 1534, 1535, 1537, 1542, 1544, 1545, 1546, 1547, 1548, 1549, 1551, 1553, 1556, 1557, 1558, 1562, 1586, 1588, 1592, 1599, 1601, 1605, 1609, 1614, 1619, 1625, 1630, 1634, 1636, 1637, 1638, 1640, 1645, 1646, 1647, 1650, 1654, 1660, 1667, 1668, 1671, 1672, 1675, 1679, 1682, 1683, 1692, 1693, 1694, 1696, 1710, 1714, 1715, 1720, 1727, 1728, 1734, 1738, 1745, 1748, 1768, 1778, 1794, 1799, 1801, 1810, 1811, 1813, 1817, 1818, 1824, 1826, 1831, 1836, 1837, 1848, 1849, 1860, 1862, 1879, 1881, 1883, 1885, 1900, 1919, 1921, 1928, 1931, 1935, 1942, 1947, 1965, 1967, 1971, 1974, 1990, 1993, 2004, 2005, 2007, 2011, 2017, 2018, 2019, 2023, 2024, 2030, 2041, 2042, 2043, 2047, 2055, 2056, 2060, 2061, 2064, 2065, 2068, 2069, 2079, 2086, 2088, 2090, 2092, 2093, 2097, 2099, 2100, 2104, 2115, 2120, 2123, 2130, 2153, 2154, 2158, 2170, 2178, 2179, 2180, 2187, 2189, 2192, 2205, 2206, 2213, 2220, 2227, 2229, 2232, 2234, 2247, 2249, 2332, 2369, 2383, 2391, 2396, 2409, 2415, 2429, 2495, 2497, 2498, 2520, 2548, 2562, 2599, 2637, 2651, 2672, 2734, 2744, 2908, 3309, 3322 and 3751.
- Malay: All sentences except sentences 1371, 1446, 1459, 2030, 2092, 2154, 2800, 2812, 3103, 3107, 3156, 3241, 3268, 3336, 3342, 3516, 3578, 3730, 3731, 3811, 3840, 3869 and 3888.
- Burmese: All sentences.
The Malay and Indonesian sentences were tokenized manually by Hiroki Nomoto and David Moeljadi, respectively. All clitics (i.e. -nya, -lah, -kah) were tokenized. In addition, the instances of the prefix se- were tokenized if they were cardinal numerals. Note that the suffix -nya and the non-numeral instances of se- were not tokenized. The following dictionaries were consulted when it was not immediately obvious whether a word sequence constituted a multiword expression.
- KBBI5. 2016. Kamus Besar Bahasa Indonesia (edisi kelima). Jakarta: Badan Pengembangan dan Pembinaan Bahasa.
- Nomoto, Hiroki. 2016. Pootaburu Nichi-Maree-Ei, Maree-Nichi-Ei Jiten [Japanese-Malay-English, Malay-Japanese-English Portable Dictionary]. Tokyo: Sanshusha.
The sentences were tokenized using the tokenize
function of Deepcut and then checked by Sunisa Wittayapanyanon and Yuka Sato. The principle adopted for the manual correction is:
- Tokenize a sequence consisting of two or more syllables if and only if all constituent syllables have a meaning that contributes to the meaning of the whole phrase/sentence.
- Do not tokenize a sequence if it contains a meaningless syllable.
- Do not tokenize a sequence if tokenizing it will change the meaning of the whole phrase/sentence.
The sentences were tokenized using the word_tokenize
function of the Undersea - Vietnamese NLP Project and then checked by Junta Nomura and Hiroki Nomoto. The following dictionary was consulted when it was not immediately obvious whether a syllable sequence constituted a multi-syllable expression.
- Hoàng, Phê, ed. 2003. Từ Điển Tiếng Việt. Đà Nẵng: Nhà Xuất Bản Đà Nẵng.
- See the pronoun substitute project page.
- One can modify the annotation, create a summary table and visualize the annotation by feeding the raw sentences and pronoun subtitute (prosub) annotation
.txt
files to ETA: Easy Text Annotator.