[benchmark][MOAT] Shuffle choices for MOAT

[waltsun]

Sorry to bother again.

I check the code for MOAT and find some minor differences from our original settings.

1. shuffle the choices if given
2. for outside knowledge text, LMMs should be told this is a hint

Theoretically, these should not cause significant performance changes. But setting so will make the implementation more consistent. We conduct a test ourselves with GPT4o_HIGH and the result is similar to ours in the paper.
(It is reasonable for results changing minorly every time, since the shuffle and the uncertainty of model itself)

Sorry again and thank you for your help.

[kennymckormick]

Hi, @waltsun ,
Is it possible to set a fixed random seed to improve the reproducibility?

[waltsun]

Hi, @waltsun , Is it possible to set a fixed random seed to improve the reproducibility?

Sure, thank you for your advice.
I made a commit where set a seed zero in the init function of the class, and ran GPT4o_HIGH settings for three times.
The results are 32.28, 31.74 and 31.22. The variation did drop a lot.
These results might be still different from reported ones in our paper, since our own implementation didn't set a seed. We believe the gap is reasonable.
(Actually setting seed in our original implementation will not work, since we build prompt concurrently, which makes the order of questions uncertain.)

Thank you again for your advice. This makes our work much better in reproducibility.