Add support for localai

[wilcomir]

Describe what you are trying to accomplish and why in non technical terms
I want to be able to use my localai instance with the genai features so that I can avoid paid services. Right now I am using ollama, but I would prefer localai support to be there as well

Describe the solution you'd like
Add the possibility to specify a base_url for openai:

genai:
  enabled: True
  provider: openai
  base_url: http://localhost:8080
  api_key: "{FRIGATE_LOCALAI_API_KEY}"  # optional
  model: gpt-4o

Describe alternatives you've considered
Using ollama is a possible alternative, and what I am currently doing, but localai is more flexible than ollama, and I would rather have a single stack up for everything.

Additional context
As a side note, one of the main goals of localai is being a drop-in replacement for openai APIs, this is why I am suggesting to just add a base_url option - nothing else should be necessary. I could accomplish something like this spoofing the openai domain for the frigate instance I am running, but of course I'd rather not.

In case there is little time to implement this, I can try and submit a PR - I have never worked on such a big project so it might take me a while. I decided to open the feature request before putting any actual work in, because first I want to hear your feedback on the implementation suggestion, and second this looks like something a person confident with this codebase can take care of in 1/10th of the time it would take me.

[saket424]

@wilcomir

Here is a 2 line modification that accomplishes what you are looking for

diff --git a/frigate/genai/openai.py b/frigate/genai/openai.py
index 4568905a..c2f9c77a 100644
--- a/frigate/genai/openai.py
+++ b/frigate/genai/openai.py
@@ -21,7 +21,7 @@ class OpenAIClient(GenAIClient):
 
     def _init_provider(self):
         """Initialize the client."""
-        return OpenAI(api_key=self.genai_config.api_key)
+        return OpenAI(base_url=self.genai_config.base_url, api_key=self.genai_config.api_key)
 
     def _send(self, prompt: str, images: list[bytes]) -> Optional[str]:
         """Submit a request to OpenAI."""
@@ -42,7 +42,7 @@ class OpenAIClient(GenAIClient):
                             }
                             for image in encoded_images
                         ]
-                        + [prompt],
+                        + [{"type": "text", "text":prompt }],
                     },
                 ],
                 timeout=self.timeout,

[NickM-27]

there was potential for this to be supported but the contributor did not confirm if testing was done #14953

[wilcomir]

Looks interesting! any way I can test it? I imagine I would need to clone the repo, checkout the PR branch, compile & test - would that work? I mean unless the CI/CD pipeline builds a docker image for each PR - which I doubt.

[NickM-27]

it builds a local docker image for test but not one that can be pulled.

[wilcomir]

I see. Can I manually download it? I am using the tensorrt version but I guess just for this test I can use anything...

[NickM-27]

it would be easier to just test on a local dev instance. But in general, regardless, I think at this point such a change wouldn't be brought in to 0.15, would need to wait for 0.16. Would have to see what other maintainers think though

[wilcomir]

Thanks @NickM-27 I understand. Let's see if I am able to do this in the next few days - in case it can make things easier I am willing to expose my localai instance to let you or someone else do this test - we would need to agree to some timeframe and I can provide an API key - normally the instance would not be exposed.

[saket424]

@wilcomir
You can see if asetlur13/frigate:0.15.0-87e7b62c-tensorrt works for you with localai . For me, asetlur13/frigate:0.15.0-87e7b62c works with localai

[saket424]

@NickM-27 @hawkeye217
I can confirm testing was done and base_url is optional as you desired

diff --git a/docs/docs/configuration/genai.md b/docs/docs/configuration/genai.md
index 9d5f62b8..b934e6cc 100644
--- a/docs/docs/configuration/genai.md
+++ b/docs/docs/configuration/genai.md
@@ -114,8 +114,12 @@ genai:
   provider: openai
   api_key: "{FRIGATE_OPENAI_API_KEY}"
   model: gpt-4o
+  base_url: http://example.endpoint/v1 # optional
 
+### Optional base_url Parameter
+The base_url parameter is an optional configuration setting that allows you to specify a custom endpoint for the OpenAI API. By default, the OpenAI client uses the standard API endpoint provided by OpenAI. However, if you need to route your requests through a different server or use a proxy, you can set this parameter to the desired URL.
+
 ## Azure OpenAI
 
 Microsoft offers several vision models through Azure OpenAI. A subscription is required.

diff --git a/frigate/genai/openai.py b/frigate/genai/openai.py
index 4568905a..7e7419da 100644
--- a/frigate/genai/openai.py
+++ b/frigate/genai/openai.py
@@ -21,7 +21,9 @@ class OpenAIClient(GenAIClient):
 
     def _init_provider(self):
         """Initialize the client."""
-        return OpenAI(api_key=self.genai_config.api_key)
+        if self.genai_config.base_url is None or self.genai_config.base_url == "":
+            self.genai_config.base_url = "https://api.openai.com/v1"
+        return OpenAI(base_url=self.genai_config.base_url, api_key=self.genai_config.api_key)
 
     def _send(self, prompt: str, images: list[bytes]) -> Optional[str]:
         """Submit a request to OpenAI."""
@@ -42,7 +44,7 @@ class OpenAIClient(GenAIClient):
                             }
                             for image in encoded_images
                         ]
-                        + [prompt],
+                        + [{"type": "text", "text":prompt }],
                     },
                 ],
                 timeout=self.timeout,

[wilcomir]

Thanks @saket424 - I believe the original PR is somewhat better, as now you are hardcoding the openai url in the calling code, but the url is already hardcoded in the OpenAI library. You should just pass None. Perhaps you can add your comments to the PR though, and the devs can take it from there? Nick mentioned this will need further review/approval anyway - tbh adding the base_url option to openai is a bit "hacky" on my book, they might want to explicitly support localai and equivalents through a different provider entirely - I suggested this implementation only because I think this is the fastest way to get it done, but maybe they prefer something different.