bug: Not enough memory when running on Kaggle

[tonezzz]

Description

In order to use free accelerator I'm trying to run it on https://kaggle.com with GPU T4x2 config but I've got load_model error, not enough memory. Could you help me finding some solutions?

Is it possible to reduce some memory usage?
Is it possible to use TPU? How?

Thanks,
Tony.

Expected results

Program load model & execute successfully.

Current results

Program crashed, with error message:

• Your notebook tried to allocate too much memory and could not be recovered. You will need to start a new session.
• Your notebook tried to allocate more memory than is available. It has restarted.

Steps to reproduce

Open a notebook at Kaggle.com, set options to GPU T4 x2 and run the following code.

!pip install --pre --upgrade pythainlp[full]

from pythainlp.chat.core import ChatBotModel
import torch

chatbot = ChatBotModel()
chatbot.load_model(device="cpu" ,torch_dtype=torch.bfloat16)

PyThaiNLP version

4.1.0beta2

Python version

3.10.12

Operating system and version

Linux fc5394d1b557 5.15.120+ #1 SMP Fri Jul 21 03:39:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

More info

No response

Possible solution

No response

Files

No response

[wannaphong]

Hello! You can use load_in_8bit=True in ChatBotModel.load_model.

[tonezzz]

Thanks, but there's still error.

[wannaphong]

I think you should restart kernel after install bitsandbytes.