Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support gpt-4-vision-preview #247

Open
PaarthShah opened this issue Nov 7, 2023 · 5 comments
Open

Support gpt-4-vision-preview #247

PaarthShah opened this issue Nov 7, 2023 · 5 comments
Labels
enhancement New feature or request

Comments

@PaarthShah
Copy link

PaarthShah commented Nov 7, 2023

https://platform.openai.com/docs/guides/vision

It seems like uploading base64-encoded images may be a generic viable strategy for passing images through the API.

Alternatively/for speed, and from unencrypted rooms, it may instead be possible/desirable to pass an image URL by transforming the image mxc url to an https url via the image_url key.

@max298
Copy link
Collaborator

max298 commented Nov 7, 2023

As far as I can tell we're limited by the library we use for API communication, which does not yet support vision. Although I'm very interested and will check what we can do as soon as the library adds support.

@Dual-0
Copy link

Dual-0 commented Nov 7, 2023

I open up a request.

@max298
Copy link
Collaborator

max298 commented Nov 8, 2023

I think we might consider dropping the third party SDK and switch to the official node package from openai: https://github.com/openai/openai-node#readme which seems to support vision

@PaarthShah
Copy link
Author

Going for the official node library seems like the best option for long-term sustainability and rapid adoption of new features

@bertybuttface
Copy link
Collaborator

Yes but we would then be responsible for handling context, which is fine if someone is willing to write the code.

@bertybuttface bertybuttface added the enhancement New feature or request label Jan 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants