How do i mimic the koboldcpp format , since its better for inference.

[akhil3417]

Api requests sent using gptel are :

1st request

Input: {"model": "neuralbeagle14-7b.Q4_K_M.gguf", "messages": [{"
role": "system", "content": "You are a large language model and a
 conversation partner. Respond concisely."}, {"role": "user", "
content": "hey , how you doing?"}], "stream": true, "temperature"
: 0.7}

Output: I'm doing well, thank you for asking! How about you?

2nd request

Input: {"model": "neuralbeagle14-7b.Q4_K_M.gguf", "messages": [{"
role": "system", "content": "You are a large language model and a
 conversation partner. Respond concisely."}, {"role": "user", "
content": "hey , how you doing?"}, {"role": "assistant", "content"
: "I'm doing well, thank you for asking! How about you?"}, {"role"
: "user", "content": "too cold"}], "stream": true, "temperature":
 0.7}

Output: I'm sorry to hear that the weather is too cold for your liking. Are you indoors where it's warmer or do you have plans
to stay warm and cozy?

1st request

where as using koboldcpp web-ui:

Input: {"n": 1, "max_context_length": 1600, "max_length": 120, "
rep_pen": 1.1, "temperature": 0.7, "top_p": 0.5, "top_k": 0, "
top_a": 0.75, "typical": 0.19, "tfs": 0.97, "rep_pen_range": 1024, "
rep_pen_slope": 0.7, "sampler_order": [6, 5, 4, 3, 2, 1, 0], "
memory": "", "genkey": "KCPP1235", "min_p": 0, "dynatemp_range":
 0, "presence_penalty": 0, "logit_bias": {}, "prompt": "[The
 following is an interesting chat message log between You and
 KoboldAI.]\n\nYou: Hi.\nKoboldAI: Hello.\nYou: hey\nKoboldAI:", "
quiet": true, "stop_sequence": ["You:", "\nYou ", "\nKoboldAI: "], "
use_default_badwordsids": false}

Output: How can I assist you today?

2nd request

You:

Input: {"n": 1, "max_context_length": 1600, "max_length": 120, "
rep_pen": 1.1, "temperature": 0.7, "top_p": 0.5, "top_k": 0, "
top_a": 0.75, "typical": 0.19, "tfs": 0.97, "rep_pen_range": 1024, "
rep_pen_slope": 0.7, "sampler_order": [6, 5, 4, 3, 2, 1, 0], "
memory": "", "genkey": "KCPP4016", "min_p": 0, "dynatemp_range":
 0, "presence_penalty": 0, "logit_bias": {}, "prompt": "[The
 following is an interesting chat message log between You and
 KoboldAI.]\n\nYou: Hi.\nKoboldAI: Hello.\nYou: hey\nKoboldAI:
 How can I assist you today?\nYou: how you doing?\nKoboldAI:", "
quiet": true, "stop_sequence": ["You:", "\nYou ", "\nKoboldAI: "], "
use_default_badwordsids": false}

Output: I'm an AI language model, so I don't have emotions or feelings like a human being. However, I am functioning well and ready to help you with any questions or requests you may have. What can I do
for you today?

[karthink]

@akhil3417 Can you give me some more context? Does koboldcpp offer an HTTP API with the second message format? And if so, do you want gptel to work with koboldcpp?

[akhil3417]

it does offer the api , and inference speed has gone up as well , source ,

Gptel works fine with koboldcpp using openai-api . but wanted to mimic the way koboldcpp prompts, but its not much of a priority.

but i request you to look at huggingface chat api , it would be great if gptel could support that.
here is any unofficial api , m not sure how to make it work in gptel.

Thanks for you response.

[karthink]

• I'm not familiar with koboldcpp, do you have a link to the official API documentation?
• Huggingface: Ditto, I'd like to understand the official API docs.