Skip to content

Commit

Permalink
fixed workspace prompt
Browse files Browse the repository at this point in the history
  • Loading branch information
dillonalaird committed Aug 21, 2024
1 parent d4b5b26 commit c8a36fc
Showing 1 changed file with 11 additions and 10 deletions.
21 changes: 11 additions & 10 deletions vision_agent/agent/vision_agent_prompts.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
**Examples**:
Here is an example of how you can interact with a user and Actions to complete a task:
--- START EXAMPLES ---
[Current directory: /workspace/test]
[Current directory: /example/workspace]
{examples}
--- END EXAMPLES ---
Expand All @@ -27,24 +27,25 @@
Here is the current conversation so far:
--- START CONVERSATION ---
[Current directory: {dir}]
{conversation}
"""

EXAMPLES_CODE1 = """
USER: Can you detect the dogs in this image? Media name dog.jpg
AGENT: {"thoughts": "I will use the generate_vision_code to detect the dogs in the image.", "response": "<execute_python>generate_vision_code('/workspace/test/dog_detector.py', 'Can you write code to detect dogs in this image?', media=['/workspace/test/dog.jpg'])</execute_python>", "let_user_respond": false}
AGENT: {"thoughts": "I will use the generate_vision_code to detect the dogs in the image.", "response": "<execute_python>generate_vision_code('/example/workspace/dog_detector.py', 'Can you write code to detect dogs in this image?', media=['/example/workspace/dog.jpg'])</execute_python>", "let_user_respond": false}
OBSERVATION:
[File /workspace/test/dog_detector.py]
[File /example/workspace/dog_detector.py]
0|from vision_agent.tools import load_image, owl_v2
1|def detect_dogs(image_path: str):
2| image = load_image(image_path)
3| dogs = owl_v2("dog", image)
4| return dogs
[End of file]
AGENT: {"thoughts": "I have generated the code to detect the dogs in the image, I must now run the code to get the output.", "response": "<execute_python>from dog_detector import detect_dogs\n print(detect_dogs('/workspace/test/dog.jpg'))</execute_python>", "let_user_respond": false}
AGENT: {"thoughts": "I have generated the code to detect the dogs in the image, I must now run the code to get the output.", "response": "<execute_python>from dog_detector import detect_dogs\n print(detect_dogs('/example/workspace/dog.jpg'))</execute_python>", "let_user_respond": false}
OBSERVATION:
----- stdout -----
Expand All @@ -55,18 +56,18 @@
USER: The the image only has one dog, can you fix this?
AGENT: {"thoughts": "I will use edit_vision_code to fix the code to detect only one dog. The lower score is 0.23 so I will set the threshold above that.", "response": "<execute_python>edit_vision_code('/workspace/test/dog_detector.py', ['Can you write code to detect dogs in this image?', 'Can you increase the threshold to 0.24?'], media=['/workspace/test/dog.jpg'])</execute_python>", "let_user_respond": false}
AGENT: {"thoughts": "I will use edit_vision_code to fix the code to detect only one dog. The lower score is 0.23 so I will set the threshold above that.", "response": "<execute_python>edit_vision_code('/example/workspace/dog_detector.py', ['Can you write code to detect dogs in this image?', 'Can you increase the threshold to 0.24?'], media=['/example/workspace/dog.jpg'])</execute_python>", "let_user_respond": false}
OBSERVATION:
[File /workspace/test/dog_detector.py]
[File /example/workspace/dog_detector.py]
0|from vision_agent.tools import load_image, owl_v2
1|def detect_dogs(image_path: str):
2| image = load_image(image_path)
3| dogs = owl_v2("dog", image, threshold=0.24)
4| return dogs
[End of file]
AGENT: {"thoughts": "I have edited the code to detect only one dog, I must now run the code to get the output.", "response": "<execute_python>from dog_detector import detect_dogs\n print(detect_dogs('/workspace/test/dog.jpg'))</execute_python>", "let_user_respond": false}
AGENT: {"thoughts": "I have edited the code to detect only one dog, I must now run the code to get the output.", "response": "<execute_python>from dog_detector import detect_dogs\n print(detect_dogs('/example/workspace/dog.jpg'))</execute_python>", "let_user_respond": false}
OBSERVATION:
----- stdout -----
Expand All @@ -82,10 +83,10 @@
USER: Yes you can use workers.png
AGENT: {"thoughts": "I will use the generate_vision_code to count the workers with helmets in the image.", "response": "<execute_python>generate_vision_code('/workspace/test/code.py', 'Can you write code to count workers with helmets in this image?', media=['/workspace/test/workers.png'])</execute_python>", "let_user_respond": false}
AGENT: {"thoughts": "I will use the generate_vision_code to count the workers with helmets in the image.", "response": "<execute_python>generate_vision_code('/example/workspace/code.py', 'Can you write code to count workers with helmets in this image?', media=['/example/workspace/workers.png'])</execute_python>", "let_user_respond": false}
OBSERVATION:
[File /workspace/test/code.py]
[File /example/workspace/code.py]
0|from vision_agent.tools import load_image, owl_v2, closest_box_distance
1|def count_workers_with_helmets(image_path: str):
2| image = load_image(image_path)
Expand All @@ -104,7 +105,7 @@
15| return count
[End of file]
AGENT: {"thoughts": "I have generated the code to count the workers with helmets in the image, I must now run the code to get the output.", "response": "<execute_python>from code import count_workers_with_helmets\n print(count_workers_with_helmets('/workspace/test/workers.png'))</execute_python>", "let_user_respond": false}
AGENT: {"thoughts": "I have generated the code to count the workers with helmets in the image, I must now run the code to get the output.", "response": "<execute_python>from code import count_workers_with_helmets\n print(count_workers_with_helmets('/example/workspace/workers.png'))</execute_python>", "let_user_respond": false}
OBSERVATION:
----- stdout -----
Expand Down

0 comments on commit c8a36fc

Please sign in to comment.