@@ -223,22 +223,22 @@ version of `mxnet-mkl <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/inde
223
223
Sentence Classification
224
224
+++++++++++++++++++++++
225
225
226
- +-----------+-------------------+---------------+---------------+---------+---------+-----+ ---------+
227
- | Dataset | Model | FP32 Accuracy | INT8 Accuracy | FP32 F1 | INT8 F1 | Log | Command |
228
- +===========+===================+===============+===============+=========+=========+=====+ =========+
229
- | MRPC | bert_12_768_12 | 87.01 | 87.01 | 90.97 | 90.88 | | |
230
- +-----------+-------------------+---------------+---------------+---------+---------+-----+ ---------+
231
- | SST-2 | bert_12_768_12 | 93.23 | 93.00 | | | | |
232
- +-----------+-------------------+---------------+---------------+---------+---------+-----+ ---------+
226
+ +-----------+-------------------+---------------+---------------+---------+---------+--------------------------------------------------------------------------------------------------------------- ---------+
227
+ | Dataset | Model | FP32 Accuracy | INT8 Accuracy | FP32 F1 | INT8 F1 | Command |
228
+ +===========+===================+===============+===============+=========+=========+=============================================================================================================== =========+
229
+ | MRPC | bert_12_768_12 | 87.01 | 87.01 | 90.97 | 90.88 |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_MRPC_base_mx1.6.0b20200125.sh>`__ |
230
+ +-----------+-------------------+---------------+---------------+---------+---------+--------------------------------------------------------------------------------------------------------------- ---------+
231
+ | SST-2 | bert_12_768_12 | 93.23 | 93.00 | | |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_SST_base_mx1.6.0b20200125.sh>`__ |
232
+ +-----------+-------------------+---------------+---------------+---------+---------+--------------------------------------------------------------------------------------------------------------- ---------+
233
233
234
234
Question Answering
235
235
++++++++++++++++++
236
236
237
- +-----------+-------------------+---------+---------+---------+---------+-----+ ---------+
238
- | Dataset | Model | FP32 EM | INT8 EM | FP32 F1 | INT8 F1 | Log | Command |
239
- +===========+===================+=========+=========+=========+=========+=====+ =========+
240
- | SQuAD 1.1 | bert_12_768_12 | 81.18 | 80.32 | 88.58 | 88.10 | | |
241
- +-----------+-------------------+---------+---------+---------+---------+-----+ ---------+
237
+ +-----------+-------------------+---------+---------+---------+---------+------------------------------------------------------------------------------------------------------------------- ---------+
238
+ | Dataset | Model | FP32 EM | INT8 EM | FP32 F1 | INT8 F1 | Command |
239
+ +===========+===================+=========+=========+=========+=========+=================================================================================================================== =========+
240
+ | SQuAD 1.1 | bert_12_768_12 | 81.18 | 80.32 | 88.58 | 88.10 |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_squad1.1_base_mx1.6.0b20200125.sh>`__ |
241
+ +-----------+-------------------+---------+---------+---------+---------+------------------------------------------------------------------------------------------------------------------- ---------+
242
242
243
243
For all model settings above, we use a subset of evaluation dataset for calibration.
244
244
0 commit comments