MPS GPU not being used when running run_localGPT.py

[Satyam7166-tech]

@PromtEngineer

System: M1 pro
Model: TheBloke/Llama-2-7B-Chat-GGML

Here is my GPU usaage when I run ingest.py (with mps enabled)

And now look at the GPU usage when I run run_localGPT.py (with mps enabled)

The spike is very thick (ignore the previous thick spike. It is denoting ingest) and happens just about 2 seconds before the LLM generates the answer.
And here is the message specifying that mps is being used.

I am a complete beginner and any input will be greatly appreciated.
Thanks

[PromtEngineer]

When you run this, is the BLAS=1?

[Satyam7166-tech]

Yes, it is.

[Satyam7166-tech]

@PromtEngineer , Do you have any updates on what I can do to make this utilize mps?

Sorry to bug you. I understand if you have some other matters to attend.

[zouy50]

Hi, I'm a beginner either. Your problem really so strange that I am confused too. But I can give you some suggestions for trying:

• you can run this with terminal not in IDE
• you can use the other model try again
• you can run model without this project, such as only run model with llamacpp
• you can change the documents, such as a simple one
• check your system settings maybe some config make this problem, for example, the power manage, the other application settings affecting

Sorry can't for helping you.

[Satyam7166-tech]

Thank you so much for your efforts.

My friend was facing the same error on his mac so I think the problem is with our setup.
Only using llama.cpp was working for us (mps usage) but connecting it to vector database is where the issues began.

Though for the past couple of days, I've been playing around with Ollama and now thankfully, it's working for my use case.
Again, thanks for your efforts.

Do you think I should close the issue now? I am not sure of the protocol.

[PromtEngineer]

@Satyam7166-tech The embedding model is also using GPU and since an NVIDIA GPU is not present on your machine, that might slow down the usage of Mac GPU. Let's keep this issue open in case if someone finds a solution.