Vision Agent v3

[dillonalaird]

A couple of pros and cons with Data Interpreter and Agent Coder lead to this design.

Data Interpreter Cons:

• The subtasks are too small, it will take tasks that GPT-4o can handle fine, and subdivide them into 5-7 smaller tasks that are unnecessarily small
• When the subtasks are small, it tends to overwrite correct code a lot. For example it will write correct code for subtask 1, and then remove it by subtask 5

Data Interpreter Pros:

• Tool recommendation helps scale to more tools
• Long term memory helps the agent framework "learn" to some degree
• Planning helps with tool choice and task decomposition if it's executed in the correct way

Agent Coder Pros:

• Does very well on coding tasks (first place on human eval)

Agent Coder Cons:

• Can't plan long term
• No tool recommendation, needs all tools at once
• Nothing like long term memory to help it "learn"

This version of vision agent basically keeps planning, but does the entire plan in one shot using the Agent Coder framework. It uses the plan to do tool recommendation and also allows for long term memory lookup. For planning beyond the initial plan it will do reflection and see if it needs to execute an additional plan.

[shankar-vision-eng]

Looks good. Left very minor comment wherever the prompt was not very clear.

[shankar-vision-eng]

What is the difference between TEST and SIMPLE_TEST ? When do we use one over other ?

[dillonalaird]

Good question, so TEST is based off of the original Agent Coder, but the problem is it requires the tester to be very thorough but for these vision tasks you can't really test a lot of the edge cases (like test on images with out a person, or where the text for OCR is in a different spot), so it tends to hallucinate file names (even after I tell it specifically not to). So I made a SIMPLE_TEST that just asks it to test basic functionality. This seems to be working better on the pool case

[shankar-vision-eng]

Just went thru the code again, i see that its not being used and only used in v1 (agent coder) and v2. Should we remove it from this file then ?

[shankar-vision-eng]

Will leave it in the file as of now, we can remove it later

[shankar-vision-eng]

LGTM