Wrong reported trajectories with parallel seeds in xray

[imenelydiaker]

I have a custom benchmark that uses few tasks from WorkArena, it supports parallel seeds (example of a run in screenshot below).

Seeds and runs:

xray shows the same trajectory for both runs with seed 10, although one succeeded the other failed (the temperature for my calls is not equal to zero):

[ThibaultLSDC]

Hello @imenelydiaker, I have observed this behavior before, but am unable to reproduce it consistently.
So far I believe it's a gradio issue with dataframes not being loaded properly. Especially experiments not being changed.
From my experience, refreshing the page when changing experiments/agents minimizes such issues.

Sorry we can't be of more help. If you find a way to make this reproducible please tell us so we can try and fix it.
Thank you :)