I can not reproduce the result JMMMU evaluation result of SakanaAI/Llama-3-EvoVLM-JP-v2

[CHENSSR]

May I ask what generation parameters you used for JMMMU evaluation result of SakanaAI/Llama-3-EvoVLM-JP-v2 except from 'A maximum output length is set to 1,024 and a temperature is set to 0 for all models during inference.'? The best accuracy I can get so far has only 28.48%. Thank you!

Groups	Version	Filter	n-shot	Metric		Value		Stderr
jmmmu_all	N/A	none		jmmmu_acc	↑	0.2848	±	N/A
- culture_agnostic	N/A	none		jmmmu_acc	↑	0.2889	±	N/A
- culture_specific	N/A	none		jmmmu_acc	↑	0.2800	±	N/A

[kcz358]

I think for this you might need to contact the author of the JMMMU or EvoVLM-JP-v2 and see if they have some advise on this