Model Performance Check

[Luodian]

Hi we open this discussion for the users of lmms-eval to have discussion about the reproductability of different models. You could post messages about any model performance mismatch with desired scores, let us know and we would try our best to assist.

[tjsgh531]

I conducted MMMU evaluation of Qwen2.5-VL-7B model, and there is a big difference in score.

The score released by open-compass/VLMEvalKit is 58.0, and the score evaluated using lmms-eval is 51.56.

Can you confirm about this?