[Howto]: Mitigate the "INFO: Received incomplete JSON response from LLM. Asking for the rest..." Loop?

[windowshopr]

Version

Command-line (Python) version

Operating System

Windows 10

Your question

Running local Ollama server using llama3 model but have also tested llama2 and mistral, the same issue persists.

I've created a new project, and am receiving a never ending loop of incomplete json response:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024-05-11 22:21:52,713 [llm_connection.py:421 - stream_gpt_completion() ] INFO: > Request model: llama3
2024-05-11 22:23:16,649 [llm_connection.py:561 - stream_gpt_completion() ] INFO: <<<<<<<<<< LLM Response <<<<<<<<<<

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024-05-11 22:23:16,649 [llm_connection.py:238 -              wrapper() ] INFO: Received incomplete JSON response from LLM. Asking for the rest...
2024-05-11 22:23:16,672 [utils.py:42 -           get_prompt() ] INFO: Getting prompt for utils/incomplete_json.prompt
2024-05-11 22:23:16,672 [llm_connection.py:421 - stream_gpt_completion() ] INFO: > Request model: llama3
2024-05-11 22:23:58,369 [llm_connection.py:561 - stream_gpt_completion() ] INFO: <<<<<<<<<< LLM Response <<<<<<<<<<

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024-05-11 22:23:58,369 [llm_connection.py:238 -              wrapper() ] INFO: Received incomplete JSON response from LLM. Asking for the rest...
2024-05-11 22:23:58,396 [utils.py:42 -           get_prompt() ] INFO: Getting prompt for utils/incomplete_json.prompt
2024-05-11 22:23:58,396 [llm_connection.py:421 - stream_gpt_completion() ] INFO: > Request model: llama3
2024-05-11 22:24:34,387 [llm_connection.py:561 - stream_gpt_completion() ] INFO: <<<<<<<<<< LLM Response <<<<<<<<<<

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024-05-11 22:24:34,387 [llm_connection.py:238 -              wrapper() ] INFO: Received incomplete JSON response from LLM. Asking for the rest...
2024-05-11 22:24:34,412 [utils.py:42 -           get_prompt() ] INFO: Getting prompt for utils/incomplete_json.prompt
2024-05-11 22:24:34,412 [llm_connection.py:421 - stream_gpt_completion() ] INFO: > Request model: llama3
2024-05-11 22:25:22,204 [llm_connection.py:561 - stream_gpt_completion() ] INFO: <<<<<<<<<< LLM Response <<<<<<<<<<

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024-05-11 22:25:22,204 [llm_connection.py:238 -              wrapper() ] INFO: Received incomplete JSON response from LLM. Asking for the rest...
2024-05-11 22:25:22,230 [utils.py:42 -           get_prompt() ] INFO: Getting prompt for utils/incomplete_json.prompt
2024-05-11 22:25:22,230 [llm_connection.py:421 - stream_gpt_completion() ] INFO: > Request model: llama3
2024-05-11 22:26:02,435 [llm_connection.py:561 - stream_gpt_completion() ] INFO: <<<<<<<<<< LLM Response <<<<<<<<<<

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024-05-11 22:26:02,435 [llm_connection.py:238 -              wrapper() ] INFO: Received incomplete JSON response from LLM. Asking for the rest...
2024-05-11 22:26:02,459 [utils.py:42 -           get_prompt() ] INFO: Getting prompt for utils/incomplete_json.prompt
2024-05-11 22:26:02,459 [llm_connection.py:421 - stream_gpt_completion() ] INFO: > Request model: llama3

This is just a small output, I've let this thing run for about an hour, same issue. There's no output to see exactly what the model is producing for a response either, so I guess question one would be how can I enable verbose?

I used an OpenAI API key and the program did work, however I don't want to pay to use their API anymore and want to use the local solutions.

I've attempted increasing the MAX_TOKENS parameter in the .env file too, same issue. How to debug?

[techjeylabs]

hey there, we have explained in our wiki how you could connect a local LLM.

You can find more about this topic here:

https://github.com/Pythagora-io/gpt-pilot/wiki/Using-GPT%E2%80%90Pilot-with-Local-LLMs

Also i invite you to join our discord community. Custom LLM topics can be discussed there.

https://discord.gg/EP9CgsVR

besides that we officially only support the pythagora subscription model and openai by now. It is simply impossible to support all models at the same time since prompts might differ. Therefore i am closing this question.

[windowshopr]

Will check it out. Thanks very much!