Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add DINOv as a new tool #44

Merged
merged 32 commits into from
Apr 19, 2024
Merged

Add DINOv as a new tool #44

merged 32 commits into from
Apr 19, 2024

Conversation

humpydonkey
Copy link
Collaborator

Example usage

from vision_agent.tools.tools import DINOv


img_path = "/Users/asia/Downloads/data/bags.jpg"
request = {
    "prompt": [
        {
            "mask": "/Users/asia/Downloads/data/mask_prompt_bags0.jpg",
            "image": img_path,
        },
        {
            "mask": "/Users/asia/Downloads/data/mask_prompt_bags1.jpg",
            "image": img_path,
        },
    ],
    "image": img_path,
}
res = DINOv()(**request)
res

AsiaCao and others added 30 commits April 9, 2024 09:42
@AsiaCao
Add DINOv as a new tool
44feb20
@AsiaCao
Fix lint errors
b18eabe
@AsiaCao
Update docs
05b1ad5
@humpydonkey @AsiaCao @dillonalaird
Fix param name mismatch (#45)
cc2035c 
Co-authored-by: Yazhou Cao <[email protected]>
@cmaloney111 @dillonalaird
Grammar/Spelling fixes (#46)
5ace291 
* Update prompts.py

* Update vision_agent_prompts.py

* Update reflexion_prompts.py

* Update vision_agent_prompts.py

* Update easytool_prompts.py

* Update prompts.py

* Update vision_agent_prompts.py
@humpydonkey @AsiaCao
@shankar-vision-eng @dillonalaird
Switch to the tools endpoint (#40)
0cd57ef 
* get endpoint ready for demo

fixed tools.json

Update vision_agent/tools/tools.py

Bug fixes

* Fix linter errors

* Fix a bug in result parsing

* Include scores in the G-SAM model response

* Removed tools.json , need to find better format

* Fixing the endpoint for CLIP and adding thresholds for grounding tools

* fix mypy errors

* fixed example notebook

---------

Co-authored-by: Yazhou Cao <[email protected]>
Co-authored-by: shankar_ws3 <[email protected]>
@humpydonkey @dillonalaird
Support streaming chat logs of an agent (#47)
073d40b 
Add a callback for reporting the chat progress of an agent
@AsiaCao @dillonalaird
Empty-Commit
dfce50b
@AsiaCao @dillonalaird
Empty-Commit: attempt to fix release
1498307
@AsiaCao @dillonalaird
[skip ci] chore(release): vision-agent 0.0.49
109eb87
@humpydonkey @AsiaCao @dillonalaird
Fix typo (#48)
32c4738 
Co-authored-by: Yazhou Cao <[email protected]>
@AsiaCao @dillonalaird
[skip ci] chore(release): vision-agent 0.0.50
a11c12d
@humpydonkey @AsiaCao @dillonalaird
Fix a typo in log (#49)
4da5d72 
Fix another typo

Co-authored-by: Yazhou Cao <[email protected]>
@AsiaCao @dillonalaird
[skip ci] chore(release): vision-agent 0.0.51
e062992
@dillonalaird @humpydonkey
Fix Baby Cam Use Case (#51)
b7cdbee 
* fix visualization error

* added font and score to viz

* changed to smaller font file

* Support streaming chat logs of an agent (#47)

Add a callback for reporting the chat progress of an agent

* fix visualize score issue

* updated descriptions, fixed counter bug

* added visualize_output

* make feedback more concrete

* made naming more consistent

* replaced individual calc ops with calculator tool

* fix random colors

* fix prompts for tools

* update reflection prompt

* update readme

* formatting fix

* fixed mypy errors

* fix merge issue

---------

Co-authored-by: Asia <[email protected]>
@AsiaCao @dillonalaird
[skip ci] chore(release): vision-agent 0.0.52
ec1a73b
@shankar-vision-eng @dillonalaird
Add image caption tool (#52)
fbe404c 
added image caption tool
@AsiaCao @dillonalaird
[skip ci] chore(release): vision-agent 0.0.53
9542d22
@humpydonkey @dillonalaird
refactor: switch model endpoints (#54)
248070e 
* Switch the host of model endpoint to api.dev.landing.ai
* DRY/Abstract out the inference code in tools
* Introduce LandingaiAPIKey and support loading from .env file
* Add integration tests for four model tools
* Minor tweaks/fixes
* Remove dead code
* Bump the minor version to 0.1.0
@AsiaCao @dillonalaird
[skip ci] chore(release): vision-agent 0.1.1
66217a4
@dillonalaird
Pool Demo (#53)
fc94a2e 
* visualized output/reflection to handle extract_frames_

* remove ipdb

* added json mode for lmm, upgraded gpt-4-turbo

* updated reflection prompt

* refactor to make function simpler

* updated reflection prompt, add tool usage doc

* fixed format issue

* fixed type issue

* fixed test case
@AsiaCao @dillonalaird
[skip ci] chore(release): vision-agent 0.1.2
1055aea
@humpydonkey @AsiaCao @dillonalaird
feat: allow disable motion detection in frame extraction function (#55)
c439bde 
* Tweak frame extraction function

* remove default motion detection, extract at 0.5 fps

* lmm now take multiple images

* removed counter

* tweaked prompt

* updated vision agent to reflect on multiple images

* fix test case

* added box distance

* adjusted prompts

---------

Co-authored-by: Yazhou Cao <[email protected]>
Co-authored-by: Dillon Laird <[email protected]>
@AsiaCao @dillonalaird
[skip ci] chore(release): vision-agent 0.1.3
da818ad
@dillonalaird
Merge branch 'main' into add-dinov
33cdf31
@dillonalaird
doc changes
94f94ba
@dillonalaird
fixed merge issues
d1d3268
@dillonalaird
fix color issue
81f3cf3
@dillonalaird
add dinov with updated endpoint
2a69082
@dillonalaird
formatting fix
34580e1
@dillonalaird dillonalaird merged commit 7d72439 into main Apr 19, 2024
7 checks passed
@dillonalaird dillonalaird deleted the add-dinov branch April 22, 2024 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dillonalaird dillonalaird dillonalaird approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@humpydonkey @dillonalaird @AsiaCao @cmaloney111 @shankar-vision-eng