Torch not compiled with CUDA enabled

[TypeCare]

hi,
i am having a RTX MX250 2GB VRAM, i want to try to use GPU and see how it behave.
i am using the below configuration:

MODEL_ID = "TheBloke/Llama-2-7b-Chat-GGUF"
MODEL_BASENAME = "llama-2-7b-chat.Q4_K_M.gguf"

however, when i run the command with "python run_localGPT.py --device_type cuda"
i get the below error after many attempts in troubleshooting and reinstalling. e.g "conda install -c pytorch torchvision cudatoolkit=10.1 pytorch"

can you advise what is the issue here?

File "C:\ProgramData\anaconda3\Lib\site-packages\torch\nn\modules\module.py", line 1143, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\anaconda3\Lib\site-packages\torch\cuda_init_.py", line 239, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

[nekiee13]

I had same issue with 4070. Just resolved it. Installed CUDA Toolkit 11.8 on global Win11 sys, nVidia Drivers remained as required for OS (12.2). Here is procedure

How to install LocalGPT

1. Check latest stable version of Pytorch https://pytorch.org/get-started/locally/#anaconda

2. Install compatible CUDA
   https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Windows&target_arch=x86_64&target_version=11&target_type=exe_local

https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Windows&target_arch=x86_64&target_version=11&target_type=exe_local
https://www.nvidia.com/Download/Find.aspx?lang=en-us
3. Create LocalGPT folder at desired location (via Win Explorer)
D:\LLM\LocalGPT
4. Clone the repo using git:
Run Anaconda prompt and move to D:\LLM\LocalGPT
D:
cd D:\LLM\LocalGPT
Repo link: https://github.com/PromtEngineer/localGPT
git clone https://github.com/PromtEngineer/localGPT.git
5. Virtual environment management using anaconda
Create virtual environment
• Run Anaconda Prompt (from Start menu, not Cmd prompt)
• See what environments are present before adding new one - localgpt
conda env list
• Deactivate base environment, or any other active virtual environment.
conda deactivate
Note: Observe change at prompt window – (base) has been deactivated
• Through anaconda prompt move to to desired LocalGPT folder location
D:
• Create virtual environment at desired location - use prefix option
conda create -p D:\LLM\LocalGPT\localgpt
cd D:\LLM\LocalGPT\localgpt
mkdir python
cd python
install python=3.10.0
• Move to D:
cd.. && cd.. && cd..
conda activate D:\LLM\LocalGPT\localgpt
• Check is the localgpt is present at right location and active -> *
conda info --envs
• If something isnt ok, then try to repeat or modify procedure, but first
conda deactivate
conda remove D:\LLM\LocalGPT\localgpt

6. Install Pythorch
   Check compatibility for each one - CUDA, PyTorch, and Python.
   • Python 3.10.0 is compatible with CUDA 11.6 or higher and PyTorch 1.11.0 or higher.
   • CUDA 11.8 is compatible with Python 3.7-3.11 and PyTorch 1.10.0 or higher.
   • PyTorch 2.0.1 is compatible with Python 3.7-3.10 and CUDA 11.3 or higher.
   Anaconda prompt from virtual environment and run
   where nvcc
   C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\nvcc.exe
   Add the directory containing nvcc to the PATH variable to active virtual environment (D:\LLM\LocalGPT\localgpt):
   set PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin;%PATH%
   This change to the PATH variable is temporary and will only persist for the current session of the virtual environment.
   conda search cudatoolkit -c pytorch
   …
   cudatoolkit 11.3.1 h59b6b97_2 pkgs/main
   cudatoolkit 11.8.0 hd77b12b_0 pkgs/main

conda install cudatoolkit=11.8
Check if nvcc is visible inside the Virtual Environment - you should be able to run
nvcc –version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:09:35_Pacific_Daylight_Time_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

nvidia-smi
537.34 Driver Version: 537.34 CUDA Version: 12.2

Install Pytorch
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

Ensure that you have installed the CUDA version of PyTorch. If you installed PyTorch without specifying a CUDA version, you might have the CPU-only version.

To troubleshoot the availability of torch.cuda, follow these steps :
Ensure that you have installed the CUDA version of PyTorch.
Check PyTorch Version with CUDA Support:
conda list pytorch

packages in environment at d:\LLM\LocalGPT\localgpt:

Name Version Build Channel

pytorch 2.0.1 py3.10_cpu_0 pytorch
pytorch-mutex 1.0 cpu pytorch
Look for a version that mentions "cuda" (e.g., "pytorch 1.10.0 py3.8_cuda11.1_cudnn8_0").
Check NVIDIA Driver Version:
The NVIDIA driver version should be compatible with the CUDA version you have installed. You can check the NVIDIA driver version using the NVIDIA Control Panel or by running:
nvidia-smi
Check CUDA Device Availability in PyTorch: Open a Python shell and run:
Python run xxxTest.py
A. The NVIDIA driver version should be compatible with the CUDA version you have installed. You can check the NVIDIA driver version using the NVIDIA Control Panel or by running:
import torch
print(torch.cuda.device_count())

This will print the number of CUDA devices available. If it returns 0, then no CUDA devices are detected by PyTorch.
B. Check CUDA Device Name. If there's at least one CUDA device, you can check its name with:
print(torch.cuda.get_device_name(0))
#tensor = torch.randn(3, 3)
#tensor_cuda = tensor.cuda()
C. Ensure Environment Variables are Set. Sometimes, certain environment variables need to be set for PyTorch to detect CUDA. Check the values of CUDA_HOME, CUDA_PATH, and PATH to ensure they include the directories of your CUDA toolkit and cuDNN library.
Reinstall PyTorch with CUDA:
If you suspect that PyTorch was installed without CUDA support, you can reinstall it with the appropriate CUDA version:
conda uninstall pytorch
conda install pytorch torchvision torchaudio cudatoolkit=11.8 -c pytorch
Also, try with
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
7. Install all other dependencies using pip
• cd to localgpt
cd D:\LLM\LocalGPT\localgpt
pip install -r requirements.txt

8. Installing LLAMA-CPP
   LocalGPT uses LlamaCpp-Python for GGML (you will need llama-cpp-python <=0.1.76) and GGUF (llama-cpp-python >=0.1.83) models. For NVIDIA GPUs support, use cuBLAS. From Anaconda prompt execute:
   set CMAKE_ARGS=-DLLAMA_CUBLAS=on
   set FORCE_CMAKE=1
   echo %CMAKE_ARGS%
   (-DLLAMA_CUBLAS=on)
   pip install llama-cpp-python==0.1.83 –no-cache-dir
   pip uninstall llama-cpp-python

Optional pip freeze –local
nvcc –-version
nvidia-smi

python xxxTEST.py
9. Ingesting your OWN Data.
Put you files in the SOURCE_DOCUMENTS folder. Support file formats
".txt": TextLoader,
".md": TextLoader,
".py": TextLoader,
".pdf": PDFMinerLoader,
".csv": CSVLoader,
".xls": UnstructuredExcelLoader,
".xlsx": UnstructuredExcelLoader,
".docx": Docx2txtLoader,
".doc": Docx2txtLoader,

You can put multiple folders within the SOURCE_DOCUMENTS folder and the code will recursively read your files. From Anaconda activated localgpt environment, run script:
Cd
python ingest.py
If you have cuda setup on your system, You will see an output like this:

This will create a new folder called DB and use it for the newly created vector store. You can ingest as many documents as you want, and all will be accumulated in the local embeddings database. If you want to start from an empty database, delete the DB and reingest your documents.

Note: When you run this for the first time, it will need internet access to download the embedding model (default: Instructor Embedding). In the subsequent runs, no data will leave your local environment and you can ingest data without internet connection.

10. RUN CLI
    In order to chat with your documents, from Anaconda activated localgpt environment, run the following command (by default, it will run on cuda).

python run_localGPT.py --show_sources --use_history
This will load the ingested vector store and embedding model. When you run this for the first time, it will need internet connection to download the LLM (default: TheBloke/Llama-2-7b-Chat-GGUF). After that you can turn off your internet connection, and the script inference would still work. No data gets out of your local environment. You will be presented with a prompt:

Enter a query:
After typing your question, hit enter. LocalGPT will take some time based on your hardware. You will get a response like this below.

Once the answer is generated, you can then ask another question or Type exit to finish the script.

[nekiee13]

Also, git clone didnt pull all files (?!). So I downloaded zip and manually added missing ones. Requirements.txt was one of missing...