Add support for additional offline models

[sabaimran]

As designed in #402, discuss the feature requests for this proposal.

• What is the motivation for the feature?
• What kind of use cases do you imagine supporting?
• How has it been tested?

@Ellen7ions , thanks for raising the PR! If you could add some more clarity to above questions, that would be helpful.

As is, the PR wouldn't quite work as intended. Anything the user inputs needs to be supported by an additional download step on Khoj's end. It wouldn't support any LlamaV2 model out of the box, but would support any GPT4All model.

This isn't a use case we were planning to support, as there's a lot of variability in model quality and hardware requirements.

Open to hearing your thoughts!

[Ellen7ions]

@sabaimran thanks for your attention ~

• What is the motivation for the feature?
  I notice that there is a issue questioning for specifying finetuned Llama model, but sorry for that I can not find that Issue 😭. finetune model #395
  In fact, I am still not quite clear about the target users for this project. If it is aimed at a broader audience, including users who are not fully familiar with LLMs, using a fixed llama model file might be more beneficial. However, for those capable of fine-tuning their own models, they may prefer to use a higher-performing Llama model for their daily needs. Maybe we can create a basic configure page and a advanced one ?
• What kind of use cases do you imagine supporting?
  You are right! In this PR, we can only use models GPT4ALL supported. It is a limitation. If we want to support for any open-source LLMs, it may be difficult.
• How has it been tested?
  I execute pytest command, but I got a weird ERROR like this:

ERROR  - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', 'E:\\khoj\\tests\\conftest.py', WindowsPath('E:/$RECYCLE.BIN/S-1-5-2...

Is my pytest version incorrect?

[sabaimran]

1. You're very right! I do think there's a lot of demand for being able to configure a custom model. Indeed, I was leaning in the direction of having one really good model that we recommend (LlamaV2), and having some on-ramp for more advanced users who want to step into customization. This might require a good deal of thought on how we want to support it. Would you like to hop on the Discord channel and grab a contributor role? That we we can go deeper in the details.
2. It can definitely be discussed -- I have some ideas of how we might want to do it. We can either give an easy way to plugin custom, online models, or allow swapping of local models via GPT4All. The tricky bit here is that all the prompt structures can be different across various models.
3. Can you try running pip install -e '.[dev]' and try running pytest again after? It's only a dev dependency, so it might not work out of the box.

[Ellen7ions]

I notice that there are more and more issues calling for more custom configurations, e.g.

• replace OpenAI URL with a local host (Allow Custom URL for OpenAI API #407)
• Add support for specifying llama model files. #402 or finetune model #395

I believe we should carefully consider the level of configurability that this project should achieve.

[sabaimran]

I very much appreciate your thoughtfulness in how to approach this problem!

[Ellen7ions]

Thanks for your appreciation~😊I've joined the Discord channel.

[Ellen7ions]

Perhaps it is not the time to provide advanced configurable options for users, this PR is on hold for now. 🎈

[sabaimran]

Sounds good! I'm going to put the PR in a closed state for now, and let's follow-up on the deeper discussion separately.

[TomLucidor]

Definitely seconding this but at the same time what are the constraints of "plug and play"? https://github.com/nichtdax/awesome-totally-open-chatgpt nichtdax/awesome-totally-open-chatgpt#30

[debanjum]

We're moving towards configurability of offline chat models. Right now you can swap any GPT4All supported offline chat model.

Once we make the chat messages follow model specific formats (chatml vs llama2 etc.) and the chat prompts user configurable, we should be in a good state for users to configure offline chat for their needs.

Also we'll add the ability to configure the OpenAI URL to use, this should allow folks to use other OpenAI compatible servers/apps as the chat model backend.

[TomLucidor]

@debanjum probably Ollama compatibility is supported as well now, so I am very optimistic about this

[debanjum]

Hey @TomLucidor, thanks for checking in ❤️. The product and the ecosystem has improved a lot in the last year!

Yeah, Khoj can use any OpenAI compatible server (e.g Ollama, LMStudio, LiteLLM) or download any GGUF chat model from HuggingFace directly within self-hosted Khoj for chat as well.

It can also read more documents, search online, read webpages, You can do deeper research, chat with your custom agents (with their own knowledge base, tools and personality) and run automations when self-hosting.