Local LLM Functionary-7b-v2.1-GGUF and Extended OpenAI Conversation

@jekalmin

@jekalmin I am running Functionary-7b-v2.1-GGUF model via Python Bindings for llama.cpp. Here is an example of my python script successfully calling the LLM and returning the correct response for get_current_weather:

The problem is, when I issue an almost identical request via Home Assistant and your Extended OpenAI Conversation I get the following error:

Here are the Home Assistant Logs:

2024-03-16 01:27:32.831 INFO (MainThread) [custom_components.extended_openai_conversation] Prompt for /data/models/huggingface/models--meetkai--functionary-7b-v2.1-GGUF/snapshots/4386d5a19700bd0ae1582574e3de13a218fb1c8e/functionary-7b-v2.1.q4_0.gguf: [{'role': 'system', 'content': "I want you to act as smart home manager of Home Assistant.\nI will provide information of smart home along with a question, you will truthfully make correction or answer using information provided in one sentence in everyday language.\n\nCurrent Time: 2024-03-16 01:27:32.826442-04:00\n\nAvailable Devices:\n```csv\nentity_id,name,state,aliases\nweather.clusterhome,Local Weather,partlycloudy,\n```\n\nThe current state of devices is provided in available devices.\nUse execute_services function only for requested action, not for current states.\nDo not execute service without user's confirmation.\nDo not restate or appreciate what user says, rather make a quick inquiry."}, {'role': 'user', 'content': 'What is the weather in New York?’}]

2024-03-16 01:27:37.786 INFO (MainThread) [custom_components.extended_openai_conversation] Response {'id': 'chatcmpl-05d7ce5d-c3d2-4fd1-ac71-1af5266ea5d4', 'choices': [{'finish_reason': 'tool_calls', 'index': 0, 'message': {'role': 'assistant', 'function_call': {'arguments': '{}', 'name': ' get_current_weather'}, 'tool_calls': [{'id': 'call_zrdjmFbhRmjp7l9010Jvr4EP', 'function': {'arguments': '{}', 'name': ' get_current_weather'}, 'type': 'function'}]}}], 'created': 1710566857, 'model': '/data/models/huggingface/models--meetkai--functionary-7b-v2.1-GGUF/snapshots/4386d5a19700bd0ae1582574e3de13a218fb1c8e/functionary-7b-v2.1.q4_0.gguf', 'object': 'chat.completion', 'usage': {'completion_tokens': 2, 'prompt_tokens': 349, 'total_tokens': 351}}

2024-03-16 01:27:37.790 ERROR (MainThread) [custom_components.extended_openai_conversation] function ' get_current_weather' does not exist
Traceback (most recent call last):
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 196, in async_process
    query_response = await self.query(user_input, messages, exposed_entities, 0)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 384, in query
    return await self.execute_tool_calls(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 477, in execute_tool_calls
    raise FunctionNotFound(function_name)
custom_components.extended_openai_conversation.exceptions.FunctionNotFound: function ' get_current_weather' does not exist

If you inspect the above logs closely you'll see that it looks like Extended OpenAI Conversation is not passing the spec I defined for get_current_weather. Basically the tools or functions part of the payload is not being passed to the LLM I think? What am I doing wrong here? Any tips?

Btw, here are the options I'm passing into your plug-in. Notice that I am in fact defining the ' get_current_weather' spec:

https://huggingface.co/TheBloke/NexusRaven-V2-13B-GGUF

this is the only one I have had any luck calling functions, specifically the execute_services. This does work if you instruct it what to do in the prompt. I am having success even with Q2.

EDIT: Sorry misread the title, thought you were using LocalAI.

In case you were wondering, the 13b runs on my i9 14th gen with 64gb ddr5 with a response time between 5-10seconds after the initial model loading. I assume this would be greatly reduced by using a GPU.

@rfam13

@rfam13 you mind sharing what you did? did you just install localAI with that model and hooked it up with HA? was there some tweaking involved regarding the prompt format?

@FutureProofHomes

Thanks for reporting an issue.
It seems LLM tries to call get_current_weather function while your function name is get_current_weather.

In order to match get_current_weather and get_current_weather, you should change the code like below.

function_name = tool.function.name.strip()

and here like below.

function_name = message.function_call.name.strip()

Ideally, LLM doesn't contain extra space in function names, but it happens.
Since it's a workaround rather than solution, I didn't apply this to the code yet.

@FutureProofHomes

Yes, it should not break any existing code.
Although it is not ideal, as you suggested, it would help digging Local LLM.

I would probably merge this in the next release.

@FutureProofHomes impressive. what would you need history for then? isnt it constrained by the small context size anyway?

ah, got it, yeah, being able to refer to at least the last couple messages is important.

re your context, the model itself only supports 8k context size:

it should still be plenty to remember the last few messages though.. agreed

Thanks @FutureProofHomes for work done!
Please give me time to catch up where you are.

Fix response error that Extended OpenAI would give in combination wit…

hello, I wanted to mention im also interested in this. I'm using LocalAI and this model works most of the time using it as is, with this integration:

https://huggingface.co/macadeliccc/laser-dolphin-mixtral-2x7b-dpo

I'm trying to get the new Llama-3-instruct 70b to work with this intergration. It does not seem to work well, becuse I think the function calling syntax is a bit different. Anyone know how to modify this model.yaml to work with this integrations functions?

name: llama3-8b-instruct
mmap: true
parameters:
  model: huggingface://second-state/Llama-3-8B-Instruct-GGUF/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf

template:
  chat_message: |
    <|start_header_id|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}}<|end_header_id|>

    {{ if .FunctionCall -}}
    Function call:
    {{ else if eq .RoleName "tool" -}}
    Function response:
    {{ end -}}
    {{ if .Content -}}
    {{.Content -}}
    {{ else if .FunctionCall -}}
    {{ toJson .FunctionCall -}}
    {{ end -}}
    <|eot_id|>
  function: |
    <|start_header_id|>system<|end_header_id|>

    You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
    <tools>
    {{range .Functions}}
    {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
    {{end}}
    </tools>
    Use the following pydantic model json schema for each tool call you will make:
    {'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
    Function call:
  chat: |
    <|begin_of_text|>{{.Input }}
    <|start_header_id|>assistant<|end_header_id|>
  completion: |
    {{.Input}}
context_size: 8192
f16: true
stopwords:
- <|im_end|>
- <dummy32000>
- "<|eot_id|>"
usage: |
      curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
          "model": "llama3-8b-instruct",
          "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
      }'

Navigation Menu

Repository navigation

Local LLM Functionary-7b-v2.1-GGUF and Extended OpenAI Conversation #174

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Local LLM Functionary-7b-v2.1-GGUF and Extended OpenAI Conversation #174

Description

Activity

rfam13 commented on Mar 19, 2024

thisIsLoading commented on Mar 23, 2024

jekalmin commented on Mar 26, 2024

jekalmin commented on Mar 26, 2024

thisIsLoading commented on Mar 29, 2024

thisIsLoading commented on Mar 29, 2024

jekalmin commented on Apr 3, 2024

Anto79-ops commented on Apr 27, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions