my installation steps and notes on improvements:

[thekit]

I am writing this post to help new users install privateGPT at sha:fdb45741e521d606b028984dbc2f6ac57755bb88

if you're cloning the repo after this point you might already have many of these issues resolved so you won't need my workarounds.

Also please note that you will most likely need to go to huggingface to get the requantized q5 models because the old models linked on the readme.md are pre-may 2023 models at time of writing.

I am on a Razr Blade 14,
cpu: AMD Ryzen 9 5900 HX
16gb ram
Nvidia GeForce RTX 3070

before you do anything else: begin downloading the embeds. it is friggin huge.

If you start now it might be done by the time you get everything else. Only download one large file at a time so you have bandwidth to get all the little packages you will be installing in the rest of this guide. or better yet start the download on another computer connected to your wifi, and you can fetch the small packages via your phone hotspot or something.

I use freedownload manager extension for chrome to manage large file downloads
https://www.freedownloadmanager.org/

then get the embeds
https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin

Once you have the embeds and models file, make a directory called c:\ai_experiments\privateGPT\models and put them both in there.

tools and software required:

##git
If you are totally new to this whole thing you will need to get git scm
https://git-scm.com/

I also recommend tortise git for a gui experience, once you have installed git.
https://tortoisegit.org/

sourcetree can also be handy, or you can choose gitflow or git kraken depending on what you like. I only really use sourcetree if I was doing a lot of dev work with multiple branches. it is not necessary to run privateGPT
https://www.sourcetreeapp.com/

python

I downloaded python 3.10 (the older one) from the windows app store. There will be other versions of python available but this is the one that is the minimum version that can compile the project. You can try a later version of python if you like, but no earlier.

a C++ compiler if you don't already have one installed

If you encounter an error while building a wheel during the pip install process, you may need to install a C++ compiler on your computer.

For Windows 10/11
To install a C++ compiler on Windows 10/11, follow these steps:

1. Install Visual Studio 2022.
2. Make sure the following components are selected:

• Universal Windows Platform development
• C++ CMake tools for Windows

3. Download the MinGW installer from the MinGW website.
4. Run the installer and select the gcc component.

a text editor

Also if you're like me and prefer vim as your text editor. if you don't like the vim "user experience", you can install nano or notepad++ or visual studio code or sublime text but you will need a fast text editor that can handle big files. Syntax highlighting is a plus.

First install chocolatey
https://chocolatey.org/install

And then follow the instructions to get your text editor
https://community.chocolatey.org/packages/vim

Chocolatey will ask you to install with admin rights to make vim available to all users, which is normal.

You might need to reboot a couple of times after installing your tools.

make a directory to house your project.

l made the directory c:\ai_experiments because the scripts can't use shortcuts to find the files you refer to and I didn't want to type a really long path into my .env file.

using windows terminal

Inside the directory I right click and Open Terminal. then right click on the task bar to pin the windows terminal to the task bar since you will be launching it often.

In terminal remember that ctrl+ins is copy and shift+ins is paste, because if you hit ctrl+C you will kill the process in the terminal. (also if your keyboard doesn't have a separate ins key, you might have to hit ctrl+fn+del for copy and shift+fn+del to paste.)

Because I put the project in c:\ root and not in my documents i sometimes have to elevate permissions to put files in.

clone the project

So with a terminal open go to c:\ai_experiments and run the command

git clone https://github.com/imartinez/privateGPT.git

It will create a directory called c:\ai_experiments\privateGPT and populate it with the project.

get the requirements

the requirements list is now fixed and so it should just be enough to do the following

run pip install -r requirements.txt like it says to in the readme.md and resolve any errors by using pip to chase down anything that doesn't automatically install correctly.

On my windows system, the packages pdfminer.six, unstructured, extract-msg, tabulate and pandoc don't appear to be part of pip. I think their functionality is covered by other packages in the list.

setting up your environment file

Make a copy of the file c:\ai_experiments\privateGPT\example.env and rename the copy just .env

Edit the contents of .env to look like this:
PERSIST_DIRECTORY=db
MODEL_TYPE=GPT4All
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
EMBEDDINGS_MODEL_NAME=all-MiniLM-L6-v2
MODEL_N_CTX=1000

I am not entirely sure if this is correct but it works for me and gives me sensible answers.

speeding up your first training run

To ingest the state of the union on my machine it takes about an hour.

I recommend deleting the state_of_the_union.txt from the source_documents directory and in its place make your own text file with a single line in it that says something outrageous and made up about yourself by name like "Jin once received a blue hedgehog for Christmas and named it Mister Pokiesworth".

You can google the phrase to make sure it isn't in the training data.

Then in the terminal run python ingest.py

This should take less than half an hour to complete. It will create a directory called db which holds your fine tunings. If you ever want to wipe out everything and go back to baseline, you can delete the db directory and ingest again.

Once it is trained run python privateGPT.py

Wait for the script to get to the part where it says Enter a query: and then ask it "What did Jin get for Christmas?"

It may give a bunch of garbage characters and warnings and then answer "I don't know", but after that it will correctly cite the document that you made that says Jin received a blue hedgehog

If that works, right click on the source_documents directory and in the pop up context menu go to "more options" and under Tortisegit select "restore".

Use Tortisegit to restore the state_of_the_union.txt

Now you can delete the db directory and do python ingest.py again.

Once it has ingested both the state of the union and the file about your personal outrageous fact, you can run python privateGPT.py questions about the content of either file and it should show you the relevant lines in that file.

If you ask a general knowledge question like "What kind of mammal is a vole?" it will return the correct answer from the main language model, but cite a random document from your source_documents directory because the main language model doesn't record its source documents.

improvements

If things are really slow first port of call is to reduce the chunk overlap size and reduce the number of returned documents from four to two as discussed here #251

You might be able to get better performance by enabling the gpu acceleration on llama as seen in this discussion #217

here is how I configured it so it runs without errors but it is very slow

1. https://github.com/abetlen/llama-cpp-python - Install using this:
   $Env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"; $Env:FORCE_CMAKE=1; pip3 uninstall llama-cpp-python; pip3 install llama-cpp-python
   Enables the use of CUDA.
2. pip uninstall langchain; pip install langchain this brings you up to langchain 0.0.172 importantly this adds the gpu layers parameter llama-cpp: add gpu layers parameter langchain-ai/langchain#4739
3. Modify the ingest.py and privateGPT.py by adding n_gpu_layers=n argument into LlamaCppEmbeddings method so it looks like this llama=LlamaCppEmbeddings(model_path=llama_embeddings_model, n_ctx=model_n_ctx, n_gpu_layers=500)

Set n_gpu_layers=500 for colab in LlamaCpp and LlamaCppEmbeddings functions, also don't use GPT4All, it won't run on GPU.

I think this means change the model_type in the .env to LlamaCpp

#217 (comment)
https://github.com/maozdemir/privateGPT-colab/blob/main/privateGPT-colab.ipynb

good luck!

[mbristol580]

I see this

PERSIST_DIRECTORY=db LLAMA_EMBEDDINGS_MODEL=models/gpt4-x-vicuna-13B.ggml.q5_1.bin MODEL_TYPE=GPT4All MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin MODEL_N_CTX=1000

but why did you put 'LLAMA_EMBEDDING_MODEL'. ? Should it have been EMBEDDING_MODEL ?

[thekit]

I had to do that to point llamacpp at it. however after this discussion I ended up removing the reference to vicuna and going back to the default example.env file #251

I will update my guide to reflect it