Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use local vision model to replace gpt-4 turbo? #180

Closed
DDDog-WANG opened this issue Apr 12, 2024 · 5 comments
Closed

How to use local vision model to replace gpt-4 turbo? #180

DDDog-WANG opened this issue Apr 12, 2024 · 5 comments
Labels

Comments

@DDDog-WANG
Copy link

I am interested in this project, I tried a lot and find this work very well. But this seems have to use a lot token of gpt, because of screenshot processing. I tried to replace gpt by local other vision model, but not find where should I modify? where is gpt vision used in the source code?

@djkramnik
Copy link

I was perusing the codebase looking for the same answer. Afaict when it calls gpt 4 vision (or whatever other model you happen to specify) it happens here:

response = await router.acompletion(model=main_model_group, messages=messages, **parameters)
. Notice how screenshot data is sent along with perhaps a text prompt. That being said, I dunno if it will be super simple to replace your own local vision model or not.

@DDDog-WANG
Copy link
Author

I perused source code, and found I have missunderstanding the role of vision-model. Vision-model in this project is not segment or locate all element (this achevied by JS script), but just check whether anything bad happen.

@suchintan
Copy link
Contributor

suchintan commented May 3, 2024

https://github.com/Skyvern-AI/skyvern/pull/251/files

New models can be added similar to the approach here - you could try it out with hosted ollama models once #242 is implemented

Copy link

github-actions bot commented Jun 3, 2024

This issue is stale because it has been open for 30 days with no activity.

@github-actions github-actions bot added the Stale label Jun 3, 2024
Copy link

This issue was closed because it has been inactive for 14 days since being marked as stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants
@suchintan @djkramnik @DDDog-WANG and others