-
Notifications
You must be signed in to change notification settings - Fork 131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Local LLM Functionary-7b-v2.1-GGUF and Extended OpenAI Conversation #174
Comments
https://huggingface.co/TheBloke/NexusRaven-V2-13B-GGUF this is the only one I have had any luck calling functions, specifically the execute_services. This does work if you instruct it what to do in the prompt. I am having success even with Q2. EDIT: Sorry misread the title, thought you were using LocalAI. In case you were wondering, the 13b runs on my i9 14th gen with 64gb ddr5 with a response time between 5-10seconds after the initial model loading. I assume this would be greatly reduced by using a GPU. |
@rfam13 you mind sharing what you did? did you just install localAI with that model and hooked it up with HA? was there some tweaking involved regarding the prompt format? |
Thanks for reporting an issue. In order to match function_name = tool.function.name.strip() and here like below. function_name = message.function_call.name.strip() Ideally, LLM doesn't contain extra space in function names, but it happens. |
Thank you @jekalmin. I actually found this solution a few days ago and should have closed the ticket. Perhaps we could consider merging this change with main because it wouldn't harm strings that don't have a trailing whitespace. e.g. the workaround doesn't have a negative impact that breaks any existing code, right? |
Yes, it should not break any existing code. I would probably merge this in the next release. |
I got everything working by sacrificing history capability. The pydantic schema behind the Functionary model isn't exactly the same as the OpenAI payloads I guess. These are the changes I made to Commented out Stripped whitespaces and again removed @jekalmin would you like me to create a NOTE: I also had to slightly modify Functionary's validation logic as well. You can see that in ticket here: MeetKai/functionary#136 |
@jekalmin it would be really awesome if you could get this up and running on your local machine, then help figure out how to get history working. You could then claim your extension works with a fully local LLM. It can control lights, locks, run scripts, and report the status of the home. If we fine-tune this model with more HA data it could be as powerful as OpenAI. |
@FutureProofHomes impressive. what would you need history for then? isnt it constrained by the small context size anyway? |
My current context threshold is set to 13000 with this model. Should be plenty enough to handle normal conversation history, I think. Here's an example of how it behaves currently without history. In this case the model doesn't know what I mean when I say, turn "them on" because it has no historical context. |
Good catch. Will set my Extended OpenAI Conversation config to 8k. Thanks. |
FYI - I believe this is our root cause for the issue. I'm fine if we think we should close this issue. |
Thanks @FutureProofHomes for work done! |
…h the Functionary LLM. Source: fixed by FutureProofHomes in jekalmin#174 (comment)
hello, I wanted to mention im also interested in this. I'm using LocalAI and this model works most of the time using it as is, with this integration: https://huggingface.co/macadeliccc/laser-dolphin-mixtral-2x7b-dpo I'm trying to get the new Llama-3-instruct 70b to work with this intergration. It does not seem to work well, becuse I think the function calling syntax is a bit different. Anyone know how to modify this model.yaml to work with this integrations functions?
|
@jekalmin I am running Functionary-7b-v2.1-GGUF model via Python Bindings for llama.cpp. Here is an example of my python script successfully calling the LLM and returning the correct response for
get_current_weather
:The problem is, when I issue an almost identical request via Home Assistant and your Extended OpenAI Conversation I get the following error:
Here are the Home Assistant Logs:
If you inspect the above logs closely you'll see that it looks like Extended OpenAI Conversation is not passing the spec I defined for
get_current_weather
. Basically thetools
orfunctions
part of the payload is not being passed to the LLM I think? What am I doing wrong here? Any tips?Btw, here are the options I'm passing into your plug-in. Notice that I am in fact defining the ' get_current_weather' spec:
The text was updated successfully, but these errors were encountered: