Support LLava or other local vision models instead of using OpenAI GPT4-vision

[ai-agents-challenge]

Feature request

Instead of relying solely ton OpenAI's GPT4-vision for image processing, provide a locally hosted alternative, such as LLAVA.

Motivation

OpenAI often gives this error when parsing images: "Your input image may contain content that is not allowed by our safety system."

[abrichr]

Related:

• Implement MiniCPM-V-2 #637
• Explore G-LLaVA #554
• Explore LLaVA #500
• Investigate Video-LLaVA #528

[abrichr]

Related: https://community.openai.com/t/your-input-image-may-contain-content-that-is-not-allowed-by-our-safety-system-vision-api-response/653372/17

I expect that the AI is denying your request because it doesn’t know if you are trying to solve a CAPTCHA or attempting to use the AI for other purposes it has been trained to prohibit, such as driving cars or tasks beyond the capabilities of computer vision.

https://community.openai.com/t/vision-api-image-not-allowed-by-our-safety-system/679147

One thing which can help is to modify the image slightly to make it look less like a CAPTCHA.
I discovered this as a side-effect of using “set-of-marks” prompting with the vision model.

Mostly it’s “business related” information that OpenAI will refuse to OCR, like people’s names, addresses, emails, phone numbers, company names, etc. So as long as your use case doesn’t involve business info you’ll be fine, …unless/until OpenAI changes their mind and censors your use case as well.