Add Long Term Memory and Feedback

[dillonalaird]

Adding the remaining two items, long term memory and feedback, for the programming Vision Agent. Tried to make the vision agent more stateless. Calling chat_with_workflow returns a lot of stuff now so that the agent doesn't have to hold on to it as state:

working_memory is the trial and error reflections the model creates when debugging failed code. You can obtain this and use it as long term memory for future usage:

output = agent.chat_with_workflow([{"role": "user", "content": "..."}])
wm = output["working_memory"]

# can save and load it
wm.save("working_mem")
wm = va.utils.load_sim("working_mem")

# merge with existing long term memory
new_ltm = va.utils.merge_sim(wm, ltm)

# can use working memory as long term memory
agent = va.agent.VisionAgentV2(long_term_memory=new_ltm)

If a subtask in the plan fails, it will return the partially completed code and plan early. You can pass a partially completed plan/conversation back to the agent to finish:

output = agent.chat_with_workflow([{"role": "user", "content": "can you code this?"}])

output = agent.chat_with_workflow(
    [{
        "role": "user",
        "content": "can you code this?"
    }, {
        "role": "assistant",
        "content": output["code"],
    }, {
        "role": "user",
        "content": "No, can you use this library?"
    }],
    plan=output["plan"],
)

Or if you want to converse with the agent (passing the old plan back is optional and probably only useful if some part of the original plan failed). This way the chat itself stays stateless, and you can track the conversation/plan.

[shankar-vision-eng]

Had some questions

[shankar-vision-eng]

Can you describe what happens in this function ? Sim returns a df with embedding calculated on the given column. From what i see we build df from a working memory which contains desc and doc. Are the desc and doc are description and doc string of tools or are they something else ? because i was under the assumption working memory is the all the artifacts from a previous run.

[dillonalaird]

Oh yeah, for memory desc and doc are probably not the best terminology. In this case, desc is the subtask string and doc is the debug attempts at trying to build code for that subtask. For example, you might have:

desc: "load the file 'dog.png'"
doc: """
image = open('cat.png')

reflection: You opened the wrong image name, it should be 'dog.png'

image = open('dog.png'
"""

So this context could be saved as long term memory. Then the next time the agent encounters the question "load the file 'dog.png'" it could retrieve this context to help it.

[shankar-vision-eng]

LGTM