Torch not compiled with CUDA enable

[Xiaom961]

hello, Im having a rough time trying to this program works, I was using Monika AI just fine, so I wanted to try this, but Im having a hard time trying to make it works:

I follow all the steps, installed cuda , libraries etc and nothing.

G:\AI+\Drive\RenAI-Chat-v1.0>run.bat
WARNING: Ignoring invalid distribution -itsandbytes (g:\ai+\drive\renai-chat-v1.0\libs\pythonlib\lib\site-packages)
WARNING: Ignoring invalid distribution -itsandbytes (g:\ai+\drive\renai-chat-v1.0\libs\pythonlib\lib\site-packages)
Requirement already satisfied: numpy==1.23.0 in g:\ai+\drive\renai-chat-v1.0\libs\pythonlib\lib\site-packages (1.23.0)
WARNING: Ignoring invalid distribution -itsandbytes (g:\ai+\drive\renai-chat-v1.0\libs\pythonlib\lib\site-packages)
WARNING: Ignoring invalid distribution -itsandbytes (g:\ai+\drive\renai-chat-v1.0\libs\pythonlib\lib\site-packages)
WARNING: Ignoring invalid distribution -itsandbytes (g:\ai+\drive\renai-chat-v1.0\libs\pythonlib\lib\site-packages)
WARNING: Ignoring invalid distribution -itsandbytes (g:\ai+\drive\renai-chat-v1.0\libs\pythonlib\lib\site-packages)

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: G:\AI+\Drive\RenAI-Chat-v1.0\libs\pythonlib\python.exe -m pip install --upgrade pip
INFO:chatbot.model:Loading tokenizer for chatbot_models/pygmalion-6b
INFO:chatbot.model:Loading the chatbot_models/pygmalion-6b model

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
G:\AI+\Drive\RenAI-Chat-v1.0\libs\pythonlib\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
Loading checkpoint shards:   0%|                                                                 | 0/2 [00:02<?, ?it/s]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ G:\AI+\Drive\RenAI-Chat-v1.0\main.py:57 in <module>                                              │
│                                                                                                  │
│    54 model_name = CHAT_CONFIG["model_name"]                                                     │
│    55 gc.collect()                                                                               │
│    56 torch.cuda.empty_cache()                                                                   │
│ ❱  57 chat_model, tokenizer = build_model_and_tokenizer_for(model_name)                          │
│    58                                                                                            │
│    59 generation_settings = {                                                                    │
│    60 │   "max_new_tokens": CHAT_CONFIG["max_new_tokens"],                                       │
│                                                                                                  │
│ G:\AI+\Drive\RenAI-Chat-v1.0\chatbot\model.py:29 in build_model_and_tokenizer_for                │
│                                                                                                  │
│    26 │   │   for bad_word in _build_bad_words_list_for(model_name)                              │
│    27 │   ]                                                                                      │
│    28 │   logger.info(f"Loading the {model_name} model")                                         │
│ ❱  29 │   model = transformers.AutoModelForCausalLM.from_pretrained(                             │
│    30 │   │   model_name,                                                                        │
│    31 │   │   bad_words_ids=bad_words_ids,                                                       │
│    32 │   │   device_map="auto",                                                                 │
│                                                                                                  │
│ G:\AI+\Drive\RenAI-Chat-v1.0\libs\pythonlib\lib\site-packages\transformers\models\auto\auto_fact │
│ ory.py:464 in from_pretrained                                                                    │
│                                                                                                  │
│   461 │   │   │   )                                                                              │
│   462 │   │   elif type(config) in cls._model_mapping.keys():                                    │
│   463 │   │   │   model_class = _get_model_class(config, cls._model_mapping)                     │
│ ❱ 464 │   │   │   return model_class.from_pretrained(                                            │
│   465 │   │   │   │   pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs,   │
│   466 │   │   │   )                                                                              │
│   467 │   │   raise ValueError(                                                                  │
│                                                                                                  │
│ G:\AI+\Drive\RenAI-Chat-v1.0\libs\pythonlib\lib\site-packages\transformers\modeling_utils.py:247 │
│ 8 in from_pretrained                                                                             │
│                                                                                                  │
│   2475 │   │   │   │   mismatched_keys,                                                          │
│   2476 │   │   │   │   offload_index,                                                            │
│   2477 │   │   │   │   error_msgs,                                                               │
│ ❱ 2478 │   │   │   ) = cls._load_pretrained_model(                                               │
│   2479 │   │   │   │   model,                                                                    │
│   2480 │   │   │   │   state_dict,                                                               │
│   2481 │   │   │   │   loaded_state_dict_keys,  # XXX: rename?                                   │
│                                                                                                  │
│ G:\AI+\Drive\RenAI-Chat-v1.0\libs\pythonlib\lib\site-packages\transformers\modeling_utils.py:279 │
│ 4 in _load_pretrained_model                                                                      │
│                                                                                                  │
│   2791 │   │   │   │   )                                                                         │
│   2792 │   │   │   │                                                                             │
│   2793 │   │   │   │   if low_cpu_mem_usage:                                                     │
│ ❱ 2794 │   │   │   │   │   new_error_msgs, offload_index, state_dict_index = _load_state_dict_i  │
│   2795 │   │   │   │   │   │   model_to_load,                                                    │
│   2796 │   │   │   │   │   │   state_dict,                                                       │
│   2797 │   │   │   │   │   │   loaded_keys,                                                      │
│                                                                                                  │
│ G:\AI+\Drive\RenAI-Chat-v1.0\libs\pythonlib\lib\site-packages\transformers\modeling_utils.py:665 │
│ in _load_state_dict_into_meta_model                                                              │
│                                                                                                  │
│    662 │   │   │   # For backward compatibility with older versions of `accelerate`              │
│    663 │   │   │   set_module_tensor_to_device(model, param_name, param_device, **set_module_kw  │
│    664 │   │   else:                                                                             │
│ ❱  665 │   │   │   set_module_8bit_tensor_to_device(model, param_name, param_device, value=para  │
│    666 │                                                                                         │
│    667 │   return error_msgs, offload_index, state_dict_index                                    │
│    668                                                                                           │
│                                                                                                  │
│ G:\AI+\Drive\RenAI-Chat-v1.0\libs\pythonlib\lib\site-packages\transformers\utils\bitsandbytes.py │
│ :77 in set_module_8bit_tensor_to_device                                                          │
│                                                                                                  │
│    74 │   │   if value is None:                                                                  │
│    75 │   │   │   new_value = old_value.to(device)                                               │
│    76 │   │   elif isinstance(value, torch.Tensor):                                              │
│ ❱  77 │   │   │   new_value = value.to(device)                                                   │
│    78 │   │   else:                                                                              │
│    79 │   │   │   new_value = torch.tensor(value, device=device)                                 │
│    80                                                                                            │
│                                                                                                  │
│ G:\AI+\Drive\RenAI-Chat-v1.0\libs\pythonlib\lib\site-packages\torch\cuda\__init__.py:239 in      │
│ _lazy_init                                                                                       │
│                                                                                                  │
│    236 │   │   │   │   "Cannot re-initialize CUDA in forked subprocess. To use CUDA with "       │
│    237 │   │   │   │   "multiprocessing, you must use the 'spawn' start method")                 │
│    238 │   │   if not hasattr(torch._C, '_cuda_getDeviceCount'):                                 │
│ ❱  239 │   │   │   raise AssertionError("Torch not compiled with CUDA enabled")                  │
│    240 │   │   if _cudart is None:                                                               │
│    241 │   │   │   raise AssertionError(                                                         │
│    242 │   │   │   │   "libcudart functions unavailable. It looks like you have a broken build?  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AssertionError: Torch not compiled with CUDA enabled

[Rubiksman78]

I've updated the zip from the Drive very recently and there is indeed an issue with torch version for GPU. Correcting this asap and reuploading it.

You can fix it right now and it will do the same thing by running this command from a terminal opened in the RenAI folder:
"libs/pythonlib/python.exe" -m pip install --force torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117

[Xiaom961]

Ohh, I was already going crazy thinking that I messed up some file. I spent the whole afternoon looking for a solution. Now I'm heading to work, so in a few more hours, I will try to fix it.

Thank you very much for the quick response. I will let you know how it went with the problem.

[Rubiksman78]

I've updated the file, you can download the zip from the Drive again and replace your previous one.

[Xiaom961]

I have managed to fix the problem. Honestly, the new files you updated didn't work for me; I was still experiencing the same issue. However, I was able to solve it by following the steps you provided to use Huggingface models with int8: C:\Users\MyName\AppData\Local\Programs\Python\Python39\Lib\site-packages\bitsandbytes

I had to do the same thing, but in the Drive\RenAI-Chat-v1.0\libs\pythonlib\Lib\site-packages\bitsandbytes files that I downloaded (replacing files and modifying Main.py).

When I tried to do it from the C drive as you mentioned, it didn't detect CUDA. Therefore, it seems necessary to perform these steps specifically using the files from the DRIVE

Now I can run the program with Pygmalion without any problem, but the issue I have now is that occasionally, when asking something to my chatbot, the program gives me traceback errors. They occur very randomly. But atleast the chatbot works.

[Rubiksman78]

Indeed, I should have precised the potential issue with int8 if your GPU is an old generation. What is the traceback for when it crashes?

[Xiaom961]

hapenned 10 mins ago, here the full traceback:

I'm sorry, but an uncaught exception occurred.

While running game code:
File "game/char_chat.rpy", line 122, in script call
call char_get_actions from _call_char_get_actions_1
File "game/char_chat.rpy", line 137, in script call
call is_talking from _call_is_talking
File "game/char_chat.rpy", line 165, in script
if sentences_list[0] == "server_ok":
File "game/char_chat.rpy", line 165, in
if sentences_list[0] == "server_ok":
IndexError: list index out of range

-- Full Traceback ------------------------------------------------------------

Full traceback:
File "game/char_chat.rpy", line 122, in script call
call char_get_actions from _call_char_get_actions_1
File "game/char_chat.rpy", line 137, in script call
call is_talking from _call_is_talking
File "game/char_chat.rpy", line 165, in script
if sentences_list[0] == "server_ok":
File "G:\AI+\renpy\ast.py", line 2115, in execute
if renpy.python.py_eval(condition):
File "G:\AI+\renpy\python.py", line 1080, in py_eval
return py_eval_bytecode(code, globals, locals)
File "G:\AI+\renpy\python.py", line 1073, in py_eval_bytecode
return eval(bytecode, globals, locals)
File "game/char_chat.rpy", line 165, in
if sentences_list[0] == "server_ok":
File "G:\AI+\renpy\revertable.py", line 217, in getitem
rv = list.getitem(self, index)
IndexError: list index out of range

Windows-10-10.0.19041 AMD64
Ren'Py 8.0.1.22070801
RenAI Chat 1.0
Thu May 25 12:06:13 2023

[sacredbone]

Hey, unfortunately I got the same or a very similar problem

I have a RTX4060 and CUDA 11.7.0 installed, tried to create a separate environment with the same result:

warn("The installed version of bitsandbytes was compiled without GPU support. "
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ Z:\RenAIChat\RenAIChat-1.0-pc\main.py:57 in │
│ │
│ 54 model_name = CHAT_CONFIG["model_name"] │
│ 55 gc.collect() │
│ 56 torch.cuda.empty_cache() │
│ ❱ 57 chat_model, tokenizer = build_model_and_tokenizer_for(model_name) │
│ 58 │
│ 59 generation_settings = { │
│ 60 │ "max_new_tokens": CHAT_CONFIG["max_new_tokens"], │
│ │
│ Z:\RenAIChat\RenAIChat-1.0-pc\chatbot\model.py:29 in build_model_and_tokenizer_for │
│ │
│ 26 │ │ for bad_word in _build_bad_words_list_for(model_name) │
│ 27 │ ] │
│ 28 │ logger.info(f"Loading the {model_name} model") │
│ ❱ 29 │ model = transformers.AutoModelForCausalLM.from_pretrained( │
│ 30 │ │ model_name, │
│ 31 │ │ bad_words_ids=bad_words_ids, │
│ 32 │ │ device_map="auto", │
│ │
│ Z:\RenAIChat\RenAIChat-1.0-pc\libs\pythonlib\lib\site-packages\transformers\models\auto\auto_fac │
│ tory.py:464 in from_pretrained │
│ │
│ 461 │ │ │ ) │
│ 462 │ │ elif type(config) in cls._model_mapping.keys(): │
│ 463 │ │ │ model_class = _get_model_class(config, cls._model_mapping) │
│ ❱ 464 │ │ │ return model_class.from_pretrained( │
│ 465 │ │ │ │ pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, │
│ 466 │ │ │ ) │
│ 467 │ │ raise ValueError( │
│ │
│ Z:\RenAIChat\RenAIChat-1.0-pc\libs\pythonlib\lib\site-packages\transformers\modeling_utils.py:24 │
│ 78 in from_pretrained │
│ │
│ 2475 │ │ │ │ mismatched_keys, │
│ 2476 │ │ │ │ offload_index, │
│ 2477 │ │ │ │ error_msgs, │
│ ❱ 2478 │ │ │ ) = cls._load_pretrained_model( │
│ 2479 │ │ │ │ model, │
│ 2480 │ │ │ │ state_dict, │
│ 2481 │ │ │ │ loaded_state_dict_keys, # XXX: rename? │
│ │
│ Z:\RenAIChat\RenAIChat-1.0-pc\libs\pythonlib\lib\site-packages\transformers\modeling_utils.py:27 │
│ 94 in _load_pretrained_model │
│ │
│ 2791 │ │ │ │ ) │
│ 2792 │ │ │ │ │
│ 2793 │ │ │ │ if low_cpu_mem_usage: │
│ ❱ 2794 │ │ │ │ │ new_error_msgs, offload_index, state_dict_index = _load_state_dict_i │
│ 2795 │ │ │ │ │ │ model_to_load, │
│ 2796 │ │ │ │ │ │ state_dict, │
│ 2797 │ │ │ │ │ │ loaded_keys, │
│ │
│ Z:\RenAIChat\RenAIChat-1.0-pc\libs\pythonlib\lib\site-packages\transformers\modeling_utils.py:66 │
│ 5 in load_state_dict_into_meta_model │
│ │
│ 662 │ │ │ # For backward compatibility with older versions of accelerate │
│ 663 │ │ │ set_module_tensor_to_device(model, param_name, param_device, **set_module_kw │
│ 664 │ │ else: │
│ ❱ 665 │ │ │ set_module_8bit_tensor_to_device(model, param_name, param_device, value=para │
│ 666 │ │
│ 667 │ return error_msgs, offload_index, state_dict_index │
│ 668 │
│ │
│ Z:\RenAIChat\RenAIChat-1.0-pc\libs\pythonlib\lib\site-packages\transformers\utils\bitsandbytes.p │
│ y:77 in set_module_8bit_tensor_to_device │
│ │
│ 74 │ │ if value is None: │
│ 75 │ │ │ new_value = old_value.to(device) │
│ 76 │ │ elif isinstance(value, torch.Tensor): │
│ ❱ 77 │ │ │ new_value = value.to(device) │
│ 78 │ │ else: │
│ 79 │ │ │ new_value = torch.tensor(value, device=device) │
│ 80 │
│ │
│ Z:\RenAIChat\RenAIChat-1.0-pc\libs\pythonlib\lib\site-packages\torch\cuda_init.py:289 in │
│ _lazy_init │
│ │
│ 286 │ │ │ │ "multiprocessing, you must use the 'spawn' start method" │
│ 287 │ │ │ ) │
│ 288 │ │ if not hasattr(torch._C, "_cuda_getDeviceCount"): │
│ ❱ 289 │ │ │ raise AssertionError("Torch not compiled with CUDA enabled") │
│ 290 │ │ if _cudart is None: │
│ 291 │ │ │ raise AssertionError( │
│ 292 │ │ │ │ "libcudart functions unavailable. It looks like you have a broken build? │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AssertionError: Torch not compiled with CUDA enabled

I would appreciate every help to get it running.