Plugins not available without OpenAI API key & Stable Diffusion not working as expected

[K-J-VV]

What happened?

I am trying to setup entirely locally hosted AI chat:

• Ollama as LLM
• LiteLLM as Proxy for OpenAI
• LibreChat as interface
• Stable Diffusion for image generation

All the above is working. However, having issues when trying to incorporate Stable Diffusion into the mix.

Steps to Reproduce

Problem 1: Plugins not available without OpenAI API key

When this variable is not set, the Plugins are not available. I'm only able to download the Stable Diffusion plugin once I add a random API key.

If I set the API key I get both drop down options for OpenAI and Plugins. NOTE: Texting anything to the OpenAI drop down results in errors/does not work.

Problem 2: Stable Diffusion plugin not working as expected
When I go to the plugin to ask it a question, the LLM responds and it appears to try and generate an image but nothing is actually provided. I get the following:

Is there a way to use plugins without OpenAI? For example: Have Ollama/LiteLLM be set as the default LLM that then calls the Stable Diffusion URL when necessary?

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

You are bound to receive errors because your setup is not officially tested. I'll be working on tool usage for non-openai endpoints soon

[K-J-VV]

Ahh gotchya

Looking forward to that update! Hoping LibreChat can become my all-in-one UI for a full-featured self-hosted AI

[krrishdholakia]

hey @danny-avila is there something litellm-related causing this? Happy to help