Which commit matches your published work (for HumanEval-rs)?

[geekoftheweek]

I'd like to run your code as it was implemented for the original paper. Would it be possible to add a link in the README that points to the specific commit representing the code used to achieve your results as published in the Reflexion paper? Thanks!

[geekoftheweek]

Critical reading failure on my part -- I see now that you do reference how to find the original code. Closing this. Thank you!

[geekoftheweek]

Reopening, as I can't find any code in reflexion-draft that references your published results against HumanEval-rs.

[noahshinn]

Hi @geekoftheweek , you can use the run script here https://github.com/noahshinn024/reflexion/blob/main/programming_runs/run_reflexion.sh. The only changes have been refactorings on the original code. Let me know if there are any questions

[geekoftheweek]

Hmm. Perhaps you can help me understand, then, why these two files have different prompt text?

• https://github.com/noahshinn024/reflexion/blob/8f1a188b51a8d23501ca242d2b4281eb52d5616b/generators/rs_generate.py
• https://github.com/noahshinn024/reflexion/blob/49485efc8fa2514dd98edfb8427ed27ad06fb391/programming_runs/generators/rs_generate.py

[ai-nikolai]

@noahshinn @geekoftheweek thanks for this thread.

Are there any updates on the above.

Also, I am struggling to understand the results on alfworld (and how to reproduce them), as e.g. env 1-3 all seem to have been successful for you, however, it never reaches success if running the script. (Unless the order of envs is different?)

Thank you.