You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using the oobabooga API to access my local model fails to communicate successfully. I can send information to the model via the oobabooga API, and can receive it back to oogabooga.py. ( stream=true via **optional_params even when I try to set it false). The data from the llm is sent back out via the line "return response.iter_lines()"
The error occurs in utils.py - handle_openai_chat_completion_chunk() , saying the byte string (str_line ) does not have attribute delta. OK, I know what that means; so I added the classes into utils.py:
class MyDelta:
def init(self, content, role, function_call=None, tool_calls=None):
self.content = content
self.role = role
self.function_call = function_call
self.tool_calls = tool_calls
class MyChoice:
def init(self, delta, finish_reason, index):
self.delta = MyDelta(**delta) if isinstance(delta, dict) else delta
self.finish_reason = finish_reason
self.index = index
class MyDictString:
def init(self, choices, created, id, model, object):
self.choices = [MyChoice(**choice) for choice in choices]
self.created = created
self.id = id
self.model = model
self.object = object
self.system_fingerprint = "oobabooga"
and a few lines to use them:
org_line = chunk
str_line = org_line.decode('utf-8') # bytes to string
if (str_line[:5] == "data:"):
str_dict = json.loads(str_line[5:]) # string to dictionary
else:
str_dict = json.loads(str_line) # string to dictionary
str_line = MyDictString(str_dict['choices'], str_dict['created'], str_dict['id'], str_dict['model'], str_dict['object'])
and the error(s) "go away", but... yeah. This isn't right.
I have reinstalled, but no joy. Any idea how I got onto this mess? I assume "things have changed"...
Reproduce
I have installed open-interpreter twice on windows 10; I am familiar with the oobabooga API and its documentation.
python -m pip install open-interpreter. ( I have also tried python -m pip install litellm --upgrade)
NOTE: Thus I am aware of how fast code changes are being made.
I am running open-interpreter from python, and have experimented with the parameters. I am currently using:
After getting this to work it is clear to me that this is an issue with Litellm, and not open-interpreter. Basically oobabooga modified their API a couple weeks ago and Litellm last related update was 4 months ago. I need to post a PR over there.
Describe the bug
Using the oobabooga API to access my local model fails to communicate successfully. I can send information to the model via the oobabooga API, and can receive it back to oogabooga.py. ( stream=true via **optional_params even when I try to set it false). The data from the llm is sent back out via the line "return response.iter_lines()"
The error occurs in utils.py - handle_openai_chat_completion_chunk() , saying the byte string (str_line ) does not have attribute delta. OK, I know what that means; so I added the classes into utils.py:
class MyDelta:
def init(self, content, role, function_call=None, tool_calls=None):
self.content = content
self.role = role
self.function_call = function_call
self.tool_calls = tool_calls
class MyChoice:
def init(self, delta, finish_reason, index):
self.delta = MyDelta(**delta) if isinstance(delta, dict) else delta
self.finish_reason = finish_reason
self.index = index
class MyDictString:
def init(self, choices, created, id, model, object):
self.choices = [MyChoice(**choice) for choice in choices]
self.created = created
self.id = id
self.model = model
self.object = object
self.system_fingerprint = "oobabooga"
and a few lines to use them:
and the error(s) "go away", but... yeah. This isn't right.
I have reinstalled, but no joy. Any idea how I got onto this mess? I assume "things have changed"...
Reproduce
I have installed open-interpreter twice on windows 10; I am familiar with the oobabooga API and its documentation.
python -m pip install open-interpreter. ( I have also tried python -m pip install litellm --upgrade)
NOTE: Thus I am aware of how fast code changes are being made.
I am running open-interpreter from python, and have experimented with the parameters. I am currently using:
from interpreter import interpreter
interpreter.offline = True
interpreter.verbose = False
interpreter.llm.model = "oobabooga/TheBloke_CodeLlama-7B-Python-GPTQ_gptq-4bit-32g-actorder_True"
interpreter.llm.api_base = "http://127.0.0.1:5000"
interpreter.llm.api_key = None
interpreter.llm.api_version = '2.0.2'
interpreter.llm.temperature = 0
interpreter.llm.context_window = 4096
interpreter.llm.max_tokens = 1024
interpreter.chat()
Expected behavior
a dialog that would allow me to create and run simple python code locally on my computer
Screenshots
No response
Open Interpreter version
Version: 0.2.4
Python version
Python 3.11.4
Operating System name and version
Windows 10
Additional context
No response
The text was updated successfully, but these errors were encountered: