all pdf talk llm vs pdfgear

[kalle07]

may i can ask, why is pdfgear(300MB small) such as good vs all others ?
like yours, like private-gpt, like gpt4all ...
anny idea ???

[kalle07]

hello ??? : D

[kalle07]

hellloooooo ???

and please can some one tell me if i talk with documents "khoj" is working 99% on GPU?
beause all ALL 5 pdf-talk github options are using CPU and its such as slow 2min wait for one answer!

i7-8Core and RTX4060(16GB)

[debanjum]

Hi @kalle07 , I'm not sure how pdfgear compares vs khoj.

What operating system have you installed Khoj on? And how have you installed Khoj (using pip or docker?)

To make self-hosted Khoj chat uses GPU you need currently need to ensure it using pip install khoj-assistant rather than the docker install method

[kalle07]

hey,

win10 and RTX GPU

at the moment your install option are more complicate than all other
pgvector (experimentaL) or docker which need WSL2 which dont work(and i dont want have it) on my machine

spy on Jan he has made it in one exe ;)

[debanjum]

Yeah, our self-hosted setup is complicated, especially on windows. It'd be great if you can share links to how others are doing it!

[kalle07]

I thought you know ^^
eg
https://docs.privategpt.dev/installation/getting-started/installation

[debanjum]

PrivateGPT is great! But Khoj is optimising for a different set of things:

1. It integrates into your existing apps/devices (like Obsidian, Emacs, Phone, Whatsapp etc.)
2. It can do internet search, draw images when configured
3. It supports multiple users from a single hosted instance like at Khoj Cloud. Using Postgres, Docker make sense for scaling Khoj as a multi-user, cloud app

Of course there is room to simplify the setup. But Khoj is currently easier to self-host if you're comfortable using the terminal. Otherwise using Khoj cloud maybe best

[kalle07]

i see .. mabye i wait some months for a simpler windows setup ;)

[debanjum]

What is your usecase for Khoj? Is it to chat with your PDFs? And did privateGPT or the other tools you mentioned not satisfy them?

[kalle07]

to talk orn instruct with models and talk with pdfs ... until now no software except oobadooga has a seed... I think it's important to realize what consequences a small change to the input has for the answer... or which consequences has different, temperatur top-p, quantization ...
at the moment LM Studio and Jan is very fast and Jan as i said work easy with pdf.
GPT4all work with pdf but extreemly slow.
oobadooga i only use if i want have a character and maybe with voice but OD has no pdf support.
privategpt work with pdf but only a few models work well and it is complicate to change the model (edit file)

[debanjum]

Thanks for the analysis on what you're looking for versus what the AI apps you've tried offer. It's super helpful.

Khoj doesn't currently allow you to set seed, top-p, top-k etc. for offline models via any UI. And even when it will allow for it, it'll be via a server side setting. Normal users shouldn't have to worry about such settings (that's the job of the app developer / server admins, IMO)

We'll be migrating to llama.cpp soon, so you'll be able to play around with a lot more models (i.e all ggml models) and it'll support GPU acceleration on more machine (AMD ROCm etc.)

As mentioned before, Khoj uses a client-server architecture to support multiple users, multiple interfaces and to abstract away the technical aspects from users (vs server admins).

But that makes the UX more complicated for single-user self-hosting folks wanting to do more technical stuff like set tokenizer, seed, temp etc. of models from a single interface.