@@ -17,14 +17,14 @@ We obtain result comparable or superior to full finetuning on the GLUE benchmark
17
17
18
18
| | Dataset/Method | RoBERTa base <br > Fine-tune (125M) | RoBERTa base <br > LoRA (0.8M) | DeBERTa XXL <br > Fine-tune (1.5B) | DeBERTa XXL <br > LoRA (4.7M) |
19
19
| ---| -------------------------| ----------------| --------------------------| -----------------| -----------------|
20
- | | MNLI (m-Acc/mm-Acc) | <b >87.6</b > | [ <b >87.5</b >±.3/<b >86.9</b >±.3] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa/roberta_base_lora_mnli.bin ) | 91.7/<b >91.9</b >| [ <b >91.9</b >±.1/<b >91.9</b >±.2] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_mnli.bin ) |
21
- | | SST2 (Acc) | 94.8 | [ <b >95.1</b >±.2] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa/roberta_base_lora_sst2.bin ) | <b >97.2</b > | [ 96.9±.2] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_sst2.bin ) |
22
- | | MRPC (Acc) | <b >90.2</b > | [ <b >89.7</b >±.7] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa/roberta_base_lora_mrpc.bin ) | 92.0 | [ <b >92.6</b >±.6] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_mrpc.bin ) |
23
- | | CoLA (Matthew's Corr) | <b >63.6</b > | [ <b >63.4</b >±1.2] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa/roberta_base_lora_cola.bin ) | <b >72.0</b > | [ <b >72.4</b >±1.1] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_cola.bin ) |
24
- | | QNLI (Acc) | 92.8 | [ <b >93.3</b >±.3] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa/roberta_base_lora_qnli.bin ) | <b >96.0</b > | [ <b >96.0</b >±.1] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_qnli.bin ) |
25
- | | QQP (Acc) | <b >91.9</b > | [ 90.8±.1] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa/roberta_base_lora_qqp.bin ) | 92.7 | [ <b >92.9</b >±.1] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_qqp.bin ) |
26
- | | RTE (Acc) | 78.7 | [ <b >86.6</b >±.7] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa/roberta_base_lora_rte.bin ) | 93.9 | [ <b >94.9</b >±.4] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_rte.bin ) |
27
- | | STSB (Pearson/Spearman Corr) | 91.2 | [ <b >91.5</b >±.2/<b >91.3</b >±.2] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa/roberta_base_lora_stsb.bin ) | <b >92.9</b >/92.6| [ <b >93.0</b >±.2/<b >92.9</b >±.3] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_stsb.bin ) |
20
+ | | MNLI (m-Acc/mm-Acc) | <b >87.6</b > | [ <b >87.5</b >±.3/<b >86.9</b >±.3] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa-base /roberta_base_lora_mnli.bin ) | 91.7/<b >91.9</b >| [ <b >91.9</b >±.1/<b >91.9</b >±.2] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_mnli.bin ) |
21
+ | | SST2 (Acc) | 94.8 | [ <b >95.1</b >±.2] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa-base /roberta_base_lora_sst2.bin ) | <b >97.2</b > | [ 96.9±.2] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_sst2.bin ) |
22
+ | | MRPC (Acc) | <b >90.2</b > | [ <b >89.7</b >±.7] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa-base /roberta_base_lora_mrpc.bin ) | 92.0 | [ <b >92.6</b >±.6] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_mrpc.bin ) |
23
+ | | CoLA (Matthew's Corr) | <b >63.6</b > | [ <b >63.4</b >±1.2] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa-base /roberta_base_lora_cola.bin ) | <b >72.0</b > | [ <b >72.4</b >±1.1] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_cola.bin ) |
24
+ | | QNLI (Acc) | 92.8 | [ <b >93.3</b >±.3] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa-base /roberta_base_lora_qnli.bin ) | <b >96.0</b > | [ <b >96.0</b >±.1] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_qnli.bin ) |
25
+ | | QQP (Acc) | <b >91.9</b > | [ 90.8±.1] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa-base /roberta_base_lora_qqp.bin ) | 92.7 | [ <b >92.9</b >±.1] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_qqp.bin ) |
26
+ | | RTE (Acc) | 78.7 | [ <b >86.6</b >±.7] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa-base /roberta_base_lora_rte.bin ) | 93.9 | [ <b >94.9</b >±.4] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_rte.bin ) |
27
+ | | STSB (Pearson/Spearman Corr) | 91.2 | [ <b >91.5</b >±.2/<b >91.3</b >±.2] ( https://github.com/microsoft/LoRA/releases/download/RoBERTa-base /roberta_base_lora_stsb.bin ) | <b >92.9</b >/92.6| [ <b >93.0</b >±.2/<b >92.9</b >±.3] ( https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_stsb.bin ) |
28
28
29
29
Fine-tuning numbers taken from [ Liu et al. (2019)] ( https://arxiv.org/abs/1907.11692 ) and [ He et al. (2020)] ( https://arxiv.org/abs/2006.03654 ) . Please follow the instructions in ` NLU/ ` to reproduce our result.
30
30
0 commit comments