Official KoboldCpp support

[henk717]

Is your feature request related to a problem? Please describe.
KoboldCpp is not listed as a possible provider within the extention, this can give the impression it is not compatible or may mean not everything is used depending on the implementation such as called upon samplers.

Describe the solution you'd like
KoboldCpp is added to the list and if desired we can supply a list of samplers the API supports / work together on this.

Additional context
KoboldAI has its own API, but for your project I assume sticking to the chat completions emulation is easier. If you test this yourself use the Chat Completions Adapter option in our UI to select the correct chat completions template for the model you are using, this will make it behave correctly. If there is something you need from our side let me know.

[rjmacarthy]

Hi thanks for the report. I see that koboldcpp mentions OpenAI compatible spec API is not recommended which is a shame. On the future I am going to be updating this project to only support OpenAI OpenAPI specification so it will support all providers which adhere, it's just easier in the long run. I am also planning to use a third party library called token.js which hides all complexity for a bunch of providers, unfortunately kobold isn't one there either.

Many thanks.

[henk717]

You can ignore that part, theres such a large ecosystem that uses the OpenAI API that most features are also implemented in that one. You can expect the OpenAI emulation to work long term, and many of the generate endpoint features exist in both since they share the underlying backend. They are just different entry points. Token.js should work as "OpenAI Compatible".

The main reason for the KoboldAI API to be recommended is that it gives the project freedom to do things even if they can't be done in an OpenAI compatible way, so developers specifically targeting KoboldCpp are recommended to use its own API. In your case I agree OpenAI emulation makes more sense but for users to know its properly compatible it still needs to be listed.

I don't know what the selection changes inside twinny, but if you need info on what the OpenAI side does / doesn't support I can help.