ubuntu 22.04 installation in an anaconda environment

[rcantada]

For novices like me, here is my current installation process for Ubuntu 22.04, in an anaconda environment.
There appears to be a lot of issues with cuda installation so I'm hoping this will help someone with a similar context:

OS: Ubuntu 22.04
Python environment: Anaconda3-2023.07-2-Linux-x86_64.sh
CUDA version: 11.7.0 (conda installation)
cuDNN version: 8.5.0.96 (downloaded from https://developer.nvidia.com/rdp/cudnn-archive ; I selected this cuDNN version because it appears to be the cudnn version that comes with pytorch with cuda 11.7)

test hardware
OLD CPU: AMD FX-8320E (no AVX2)
OLD GPU: 2x Quadro P4000

Please note that I removed all other installations of CUDA and cuDNNin my system as they affect the conda installation. I used a conda restricted install of cuda so as not to conflict with other anaconda environments. This would include:

• removing that installed through apt
• removing installed through the nvidia .deb or runtime

1. Create and activate a new virtual environment

conda create -n localGPT python=3.10.0
conda activate localGPT

2. Install cuda-toolkit in the conda environment

conda install -c "nvidia/label/cuda-11.7.0" cuda-toolkit

3. Copy cuDNN files to the cuda installation

extract cudnn-linux-x86_64-8.5.0.96_cuda11-archive.tar.xz
cd to the extracted folder

cp include/cudnn*.h ~/anaconda3/envs/localGPT/include
cp lib/libcudnn* ~/anaconda3/envs/localGPT/lib

4. Install pytorch with cuda support (see install matrix at https://pytorch.org/ for your context)

conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

5. Clone the repo using git

git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

6. Install the requirements

pip install -r requirements.txt

At this point I got some warning from pip:
extract-msg 0.45.0 requires olefile==0.46, which is not installed.
oletools 0.60.1 requires olefile>=0.46, which is not installed.
oletools 0.60.1 requires pyparsing<3,>=2.1.0, but you have pyparsing 3.0.9 which is incompatible.

So I installed the following packages:
pip install olefile==0.46
pip install pyparsing==2.3.1

7. Uninstall and reinstall auto-gptq from source

To resolve the following when running localGPT with GPTQ models: WARNING - qlinear_old.py:16 - CUDA extension not installed.

pip uninstall -y auto-gptq
GITHUB_ACTIONS=true pip install auto-gptq==0.2.2 --no-cache-dir

8. Install llama-cpp

CMAKE_ARGS="-DLLAMA_CUBLAS=1" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python==0.1.83 --no-cache-dir --verbose

Check the log output if the build can find your cuda installation .
Check ~/anaconda3/envs/localGPT/include if cuda files are present.

If your CPU does NOT support AVX2, disable it:

CMAKE_ARGS="-DLLAMA_CUBLAS=1 -DLLAMA_AVX2=OFF -DLLAMA_F16C=ON" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python==0.1.83 --no-cache-dir --verbose

9. Ingest data

python ingest.py

10. Ask questions to your documents, locally!

python run_localGPT.py --show_sources

If you have a better way to improve this procedure e.g. using higher cuda and cudnn versions, sharing it is appreciated.

Optional for targetting a second gpu so as not to use up your gpu for display/output
For this I prefer using the GPU UUID, so find the gpu you want to use for localGPT:

nvidia-smi -L

Create a bash script (i created mine inside the localGPT folder) e.g.

nano run_localGPT.sh

enter the following content:

#!/bin/bash

export CUDA_VISIBLE_DEVICES=<your preferred GPU UUID>
python run_localGPT.py

make the script executable then run it.

./run_localGPT.sh

there is an nvidia-smi way to monitor if the gpu is being used, but I just look at the thermals in Nvidia Xserver Settings

[rcantada]

Ingesting large files

If you get this error when ingesting large files like whole books

sqlite3.OperationalError: too many SQL variables

Lookup solutions from #489
Update: It seems that privateGPT has solved this problem (see zylon-ai/private-gpt#999). They bumped up their chromadb==0.4.12, I just can't figure out the code yet in their ingest.py .

This seems to be a problem with chromadb.
What worked for me on chromadb==0.4.6 is @ww2283 's solution below. It involves replacing the submit_embeddings() function in embeddings_queue.py . My .py file is located in /home/username/anaconda3/envs/localGPT/lib/python3.10/site-packages/chromadb/db/mixins/

# MODIFIED FOR LARGER DATA TO BE INGESTED

@override
def submit_embeddings(
    self, topic_name: str, embeddings: Sequence[SubmitEmbeddingRecord]
) -> Sequence[SeqId]:
    if not self._running:
        raise RuntimeError("Component not running")

    if len(embeddings) == 0:
        return []

    # Define the batch size
    batch_size = 500
    total_seq_ids = []

    # Split the embeddings into batches and process each batch
    for i in range(0, len(embeddings), batch_size):
        batch_embeddings = embeddings[i:i+batch_size]
        t = Table("embeddings_queue")
        insert = (
            self.querybuilder()
            .into(t)
            .columns(t.operation, t.topic, t.id, t.vector, t.encoding, t.metadata)
        )
        id_to_idx: Dict[str, int] = {}
        for embedding in batch_embeddings:
            (
                embedding_bytes,
                encoding,
                metadata,
            ) = self._prepare_vector_encoding_metadata(embedding)
            insert = insert.insert(
                ParameterValue(_operation_codes[embedding["operation"]]),
                ParameterValue(topic_name),
                ParameterValue(embedding["id"]),
                ParameterValue(embedding_bytes),
                ParameterValue(encoding),
                ParameterValue(metadata),
            )
            id_to_idx[embedding["id"]] = len(id_to_idx)
        with self.tx() as cur:
            sql, params = get_sql(insert, self.parameter_format())
            sql = f"{sql} RETURNING seq_id, id"
            results = cur.execute(sql, params).fetchall()
            seq_ids = [cast(SeqId, None)] * len(results)
            embedding_records = []
            for seq_id, id in results:
                seq_ids[id_to_idx[id]] = seq_id
                submit_embedding_record = batch_embeddings[id_to_idx[id]]
                embedding_record = EmbeddingRecord(
                    id=id,
                    seq_id=seq_id,
                    embedding=submit_embedding_record["embedding"],
                    encoding=submit_embedding_record["encoding"],
                    metadata=submit_embedding_record["metadata"],
                    operation=submit_embedding_record["operation"],
                )
                embedding_records.append(embedding_record)
            self._notify_all(topic_name, embedding_records)
            total_seq_ids.extend(seq_ids)

    return total_seq_ids

You can diff your embeddings_queue.py with this attached file. Please note that this is from chromadb==0.4.6, so if your chromadb is the newer 0.4.12, the file will be much different.

embeddings_queue.zip

The only problem I have with this solution is that everytime chromadb is reinstalled or updated, the embeddings_queue.py file will need to be modified again.

Converting pdf, epub, and mobi to text

I manually convert open access pdf, epub, and mobi to text first rather than relying on ingest.py because some files will have erroneous text flow and will require to be ocr'ed again.

Using pandoc I just run the following on the source folder:

for file in *.pdf; do pandoc -f epub -t plain -o "$file.txt" "$file"; done
for file in *.epub; do pandoc -f epub -t plain -o "$file.txt" "$file"; done
for file in *.mobi; do ebook-convert "$file" "$file.txt"; done

Warning in running run_localGPT_API.py

If you used ingest.py to manually ingest your sources and use the terminal-based run_localGPT.py, DO NOT use the webui run_localGPT_API.py as it seems to reset the DB.
I lost my DB from five hours of ingestion (I forgot to back it up) because of this. And I do not know if this has been resolved already.