slow, and random result size when asking for a list of 50 items

[dportabella]

I am testing the latest localGPT version, with the defaults (original SOURCE_DOCUMENTS, ingest.py and run_localGPT.py). No other documents ingested. The only difference, in run_localGPT.py I changed max_ctx_size and max_length from 2048 to 20480.

I have a machine with proxmox (server virtualization). Inside proxmox, I have a VM with 24 GB mem, 20 processors (1 socket, 20 cores), and NVDIA GeForce RTX 4070 with 16 GB VRAM (directly attached from host to the VM). Is this a good setup for fast replies?

I ask localGPT the following question: "tell me a list of 50 influential books. Write a json array. each item in the array is a object with the fields name, author, ISBN."

It takes 3 minutes to answer. Why is it so slow?

Also, it sometimes returns 5 books, and sometimes it returns 20 books (I ask for 50 books).
Why so?

$ python run_localGPT.py -s
2023-08-26 00:06:21,882 - INFO - run_localGPT.py:180 - Running on: cpu
2023-08-26 00:06:21,882 - INFO - run_localGPT.py:181 - Display Source Documents set to: True
2023-08-26 00:06:21,949 - INFO - SentenceTransformer.py:66 - Load pretrained SentenceTransformer: hkunlp/instructor-large
load INSTRUCTOR_Transformer
max_seq_length  512
2023-08-26 00:06:23,479 - INFO - __init__.py:88 - Running Chroma using direct local API.
2023-08-26 00:06:23,481 - WARNING - __init__.py:43 - Using embedded DuckDB with persistence: data will be stored in: /home/david/localGPT/DB
2023-08-26 00:06:23,484 - INFO - ctypes.py:22 - Successfully imported ClickHouse Connect C data optimizations
2023-08-26 00:06:23,485 - INFO - json_impl.py:45 - Using python library for writing JSON byte strings
2023-08-26 00:06:23,605 - INFO - duckdb.py:460 - loaded in 72 embeddings
2023-08-26 00:06:23,606 - INFO - duckdb.py:472 - loaded in 1 collections
2023-08-26 00:06:23,606 - INFO - duckdb.py:89 - collection with name langchain already exists, returning existing collection
2023-08-26 00:06:23,606 - INFO - run_localGPT.py:45 - Loading Model: TheBloke/Llama-2-7B-Chat-GGML, on: cpu
2023-08-26 00:06:23,606 - INFO - run_localGPT.py:46 - This action can take a few minutes!
2023-08-26 00:06:23,606 - INFO - run_localGPT.py:50 - Using Llamacpp for GGML quantized models
llama.cpp: loading model from /home/david/.cache/huggingface/hub/models--TheBloke--Llama-2-7B-Chat-GGML/snapshots/b616819cd4777514e3a2d9b8be69824aca8f5daf/llama-2-7b-chat.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 20480
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =    0.07 MB
llama_model_load_internal: mem required  = 5407.71 MB (+ 1026.00 MB per state)
llama_new_context_with_model: kv self size  = 10240.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 

Enter a query: tell me a list of 50 influential books. Write a json array. each item in the array is a object with the fields name, author, ISBN.

llama_print_timings:        load time =   260.78 ms
llama_print_timings:      sample time =    69.53 ms /   177 runs   (    0.39 ms per token,  2545.52 tokens per second)
llama_print_timings: prompt eval time = 40834.08 ms /  1286 tokens (   31.75 ms per token,    31.49 tokens per second)
llama_print_timings:        eval time = 17662.99 ms /   176 runs   (  100.36 ms per token,     9.96 tokens per second)
llama_print_timings:       total time = 58978.55 ms


> Question:
tell me a list of 50 influential books. Write a json array. each item in the array is a object with the fields name, author, ISBN.

> Answer:
 

books: [
{name:"The Wealth of Nations",author:"Adam Smith",ISBN:"9781465439121"},
{name:"The Communist Manifesto",author:"Karl Marx and Friedrich Engels",ISBN:"9780451524377"},
{name:"On Liberty",author:"John Stuart Mill",ISBN:"9780140126295"},
{name:"Democracy in America",author:"Alexis de Tocqueville",ISBN:"9780140150335"},
{name:"The Prince",author:"Niccolo Machiavelli",ISBN:"9780140446524"}
]