Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Claude Sonnet 3.5 VisionAgentCoder #231

Merged
merged 22 commits into from
Sep 11, 2024
Merged

Add Claude Sonnet 3.5 VisionAgentCoder #231

merged 22 commits into from
Sep 11, 2024

Conversation

dillonalaird
Copy link
Member

@dillonalaird dillonalaird commented Sep 9, 2024 •
edited
Loading

Add support for Claude Sonnet 3.5 also fix some issues:

  • Update extract_json to work on more side cases introduced with Claude Sonnet 3.5
  • Redo debugger so instead of outputting code inside the JSON it outputs it inside markdown. This is because models, particularly Claude Sonnet 3.5, struggle to generate code inside JSON format. https://aider.chat/2024/08/14/code-in-json.html
  • Fix bug in extract_frames documentation, was stating an incorrect default argument for FPS
  • Allow debugger to see the tool documentation in case the bug was a missing import
  • Add ability to print execution results without printing returned results. This is because all tool tracing gets added to the returned results which we don't want to show the model.
  • Fixes edge case on OCR where it will fail if the image has 0 height or 0 width
  • Updated plan reviewer prompt to better evaluate multiple plans

shankar-vision-eng
shankar-vision-eng reviewed Sep 10, 2024
View reviewed changes
Copy link
Collaborator

@shankar-vision-eng shankar-vision-eng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left few comments, will sync with you soon

vision_agent/agent/agent_utils.py Show resolved Hide resolved
vision_agent/agent/vision_agent_coder.py Show resolved Hide resolved
vision_agent/agent/vision_agent_coder_prompts.py Outdated Show resolved Hide resolved
vision_agent/agent/vision_agent_coder_prompts.py Outdated Show resolved Hide resolved
shankar-vision-eng
shankar-vision-eng requested changes Sep 11, 2024
View reviewed changes
Copy link
Collaborator

@shankar-vision-eng shankar-vision-eng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test case for the ixc25_temporal_localization ?

shankar-vision-eng
shankar-vision-eng approved these changes Sep 11, 2024
View reviewed changes
Copy link
Collaborator

@shankar-vision-eng shankar-vision-eng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dillonalaird dillonalaird merged commit 777b4d5 into main Sep 11, 2024
8 checks passed
@dillonalaird dillonalaird deleted the add-claude-35 branch September 11, 2024 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shankar-vision-eng shankar-vision-eng shankar-vision-eng approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dillonalaird @shankar-vision-eng