Skip to content

Latest commit

 

History

History
186 lines (140 loc) · 4.93 KB

README.md

File metadata and controls

186 lines (140 loc) · 4.93 KB
Raw

Reproduce BERT finetune results for chinese datasets

This repo reproduces BERT finetune resutlts for chinese datasets. Data can be downloaded from Baidu ERNIE repo(https://github.com/PaddlePaddle/LARK/tree/develop/ERNIE, https://ernie.bj.bcebos.com/task_data.tgz), which contains MSRA-NER, LCQMC, XNLI, ChnSentiCorp, nlpcc-dbqa.

MSRA-NER

Baidu reproduce result:

Epoch F1 in develop F1 in test
3 94.0% 92.6%

Our reproduce result:

Epoch F1 in develop F1 in test
2 94.15% 92.34%
10 95.75% 94.89%

run script:

python run_ner.py \
-device_map 0 \
-data_dir data/baidu_msra/bio/ \
-output_dir output/google_bert_wikicn_epoch2 \
-init_checkpoint bert_model/chinese_L-12_H-768_A-12/bert_model.ckpt \
-bert_config_file bert_model/chinese_L-12_H-768_A-12/bert_config.json \
-vocab_file bert_model/chinese_L-12_H-768_A-12/vocab.txt \
-batch_size 16 \
-decode=softmax \
-num_train_epochs=2

XNLI

XNLI data was download from google's bert repo.

google's result:

Epoch Acc in develop (infer from log) Acc in test
2 77.4% 77.2%

Baidu reproduce result:

Epoch Acc in develop Acc in test
3 78.1% 77.2%

Our reproduce result:

Epoch Acc in develop Acc in test
2 77.91% 77.56%

run script:

python run_classifier.py \
--task_name=XNLI \
--do_train=true \
--do_eval=true \
--data_dir=../../../corpus/xnli/XNLI-MT-1.0 \
--vocab_file=bert_model/chinese_L-12_H-768_A-12/vocab.txt \
--bert_config_file=bert_model/chinese_L-12_H-768_A-12/bert_config.json \
--init_checkpoint=bert_model/chinese_L-12_H-768_A-12/bert_model.ckpt \
--max_seq_length=128 \
--train_batch_size=32 \
--learning_rate=5e-5 \
--num_train_epochs=2.0 \
--output_dir output_model/bert_xnli_output_ep4/

LCQMC

Baidu reproduce result:

Epoch Acc in develop Acc in test
3 88.8% 87.0%

Our reproduce result:

Epoch Acc in develop Acc in test
3 88.7% 86.49%

run script:

python run_classifier.py \
--task_name=LCQMC \
--do_train=true \
--do_eval=true \
--data_dir=../../../corpus/baidu_ernie_task_data/lcqmc/ \
--vocab_file=bert_model/chinese_L-12_H-768_A-12/vocab.txt \
--bert_config_file=bert_model/chinese_L-12_H-768_A-12/bert_config.json \
--init_checkpoint=bert_model/chinese_L-12_H-768_A-12/bert_model.ckpt \
--max_seq_length=128 \
--train_batch_size=32 \
--learning_rate=5e-5 \
--num_train_epochs=2.0 \
--output_dir output_model/bert_lcqmc_output_ep3/

ChnSentiCorp

Baidu reproduce result:

Epoch Acc in develop Acc in test
3 94.6% 94.3%

Our reproduce result:

Epoch Acc in develop Acc in test
5 92.7% 94.5%
10 93.58% 94.5%

run script:

python run_classifier.py \
--task_name=chnsenticorp \
--do_train=true \
--do_eval=true \
--data_dir=../../../corpus/baidu_ernie_task_data/chnsenticorp/ \
--vocab_file=bert_model/chinese_L-12_H-768_A-12/vocab.txt \
--bert_config_file=bert_model/chinese_L-12_H-768_A-12/bert_config.json \
--init_checkpoint=bert_model/chinese_L-12_H-768_A-12/bert_model.ckpt \
--max_seq_length=128 \
--train_batch_size=32 \
--learning_rate=5e-5 \
--num_train_epochs=5.0 \
--output_dir output_model/bert_chnsenticorp_output_ep5/

nlpcc-dbqa

Baidu reproduce result:

Epoch MRR in develop MRR in test F1 in develop F1 in test
3 94.7% 94.6% 80.7% 80.8%

Our reproduce result:

Epoch MRR in develop MRR in test F1 in develop F1 in test
3 93.9% 94.0% 78.6% 81.3%

run script:

python run_classifier.py \
--task_name=nlpccdbqa \
--do_train=true \
--do_eval=true \
--do_predict=true \
--data_dir=../../../corpus/baidu_ernie_task_data/chnsenticorp/ \
--vocab_file=bert_model/chinese_L-12_H-768_A-12/vocab.txt \
--bert_config_file=bert_model/chinese_L-12_H-768_A-12/bert_config.json \
--init_checkpoint=bert_model/chinese_L-12_H-768_A-12/bert_model.ckpt \
--max_seq_length=128 \
--train_batch_size=32 \
--learning_rate=5e-5 \
--num_train_epochs=3.0 \
--output_dir output_model/bert_nlpccdbqa_output_ep2/

Reference

https://github.com/google-research/bert

https://github.com/kyzhouhzau/BERT-NER

https://github.com/macanv/BERT-BiLSTM-CRF-NER