Ingest taking very long ⌛️

[vicdotdevelop]

Dear privateGPT community,

I am running an ingest of 16 pdf documents all over 43MB of documents.
The ingest is still running but it runs already for around 7 hours.
Is there anything to do, to speed up the process or see the actual progress?
Its still doing something as it updates with news lines in the terminal all the time.

Any suggestions 🥲?

[andrewginns]

That seems incredibly slow, something like that should take minutes not hours. What are your hardware specifications?

[thekit]

I'm not OP but it took me half an hour to ingest the state of the union on a Razer Blade 14
AMD Ryzen 9 5900HX
16GB ram
windows 11 home
and about 500mb free on my m.2 nvme hdd

[imartinez]

Hey Victor. I guess you are not up to date with the latest changes on main branch. We changed to a different embeddings model that is a LOT faster (I couldn't even measure it... 100x or maybe 1000x faster). I ingested a 60 pages PDF in 10 seconds.

I'd advice to stop the ingest, pull latest changes, remove the "db" folder and start again.

[imartinez]

Share the results!

[thekit]

doing a pull and ingest caused a crash.

I will try reinstalling my prerequisites again

PS C:\Users\jinke> cd C:\ai_experiments\privateGPT
PS C:\ai_experiments\privateGPT> git reset --hard; git pull
HEAD is now at fdb4574 Merge pull request #211 from mdeweerd/extra_loaders
remote: Enumerating objects: 34, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 34 (delta 16), reused 16 (delta 9), pack-reused 6
Unpacking objects: 100% (34/34), 13.52 KiB | 147.00 KiB/s, done.
From https://github.com/imartinez/privateGPT
fdb4574..ad64589 main -> origin/main
Updating fdb4574..ad64589
Fast-forward
README.md | 24 +++++++++++++++---------
example.env | 2 +-
ingest.py | 20 ++++++++++----------
privateGPT.py | 8 ++++----
requirements.txt | 10 +++++-----
5 files changed, 35 insertions(+), 29 deletions(-)
PS C:\ai_experiments\privateGPT> python .\ingest.py
Loading documents from source_documents
Loaded 2 documents from source_documents
Split into 91 chunks of text (max. 500 characters each)
Traceback (most recent call last):
File "C:\ai_experiments\privateGPT\ingest.py", line 96, in
main()
File "C:\ai_experiments\privateGPT\ingest.py", line 87, in main
embeddings = HuggingFaceEmbeddings(model_name=embeddings_model_name)
File "C:\Users\jinke\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\langchain\embeddings\huggingface.py", line 44, in init
super().init(**kwargs)
File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for HuggingFaceEmbeddings
model_name
none is not an allowed value (type=type_error.none.not_allowed)
PS C:\ai_experiments\privateGPT>

[thekit]

I figured out what was wrong with me, I needed to do a pip install -r requirements.txt

and then I needed to change my .env to use the new embeds

PERSIST_DIRECTORY=db
MODEL_TYPE=GPT4All
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
EMBEDDINGS_MODEL_NAME=all-MiniLM-L6-v2
MODEL_N_CTX=1000

finally I had to delete the content of my db directory and run python ingest.py again

[vicdotdevelop]

Share the results!

Hey @imartinez, it worked very well :) I just pulled the latest changes and restarted the ingest - thanks for your support!

[imartinez]

I figured out what was wrong with me, I needed to do a pip install -r requirements.txt

and then I needed to change my .env to use the new embeds

PERSIST_DIRECTORY=db
MODEL_TYPE=GPT4All
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
EMBEDDINGS_MODEL_NAME=all-MiniLM-L6-v2
MODEL_N_CTX=1000

finally I had to delete the content of my db directory and run python ingest.py again

Sorry for the breaking changes... The repo is still so early and things are evolving so fast that it is impossible to achieve stability. Forcing ourselves to stabilize would reduce the speed of the progress too much.

I'm glad you figured it out!

[joru10]

I started yesterday , now running for over 24h, and not done yet, but I added over 1000 pdfs (average 100 tokens per PDF)
I would agree in general that improving the perf of this ingestion stage would be a great improvement

[joru10]

just saw imartinez response just 5 min ago... great news, I will stop it and bet on the new change...

[imartinez]

Let us know how it goes!

[joru10]

Thanks, just to get back on this, I noticed the new method, that appears to try to ingest all docs at once?
Point is that after about over 30 min on waiting, I get this error message, I tried to understand it with the code, but not trivial (not to me at least).

Best I could find is the following to do:

1. Update PDFMiner library to the latest version (I just loaded the code so I do not think this will help.
2. Modify the code in pdffont.py to handle the non-int values of 'float' type, for example, by casting the sequence to 'int' or by rounding off the 'float' values. (maybe?)
3. Use a different PDF parsing library, such as PyPDF2 or pdfplumber, which might not have this issue. (I leave it there in case this may be of interest)
4. Check if the PDF file being parsed has any irregularities in its fonts or metadata that might cause this TypeError and correct them before parsing. (possible, most files are auto-generated from OneNote pages, who knows, but I have no idea of which of the 1500 files might be causing this)
5. Use a try-except block to catch this particular TypeError and handle it gracefully, for example, by skipping the problematic pages or by logging the error message and continuing with the parsing. (yes, yes, if I would be able to track what are those files)

Below what I get from the execution (I tried it twice):

Loading documents from source_documents
Traceback (most recent call last):
File "c:\Users\jose\privateGPT\ingest.py", line 97, in
main()
File "c:\Users\jose\privateGPT\ingest.py", line 81, in main
documents = load_documents(source_directory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\jose\privateGPT\ingest.py", line 68, in load_documents
return [load_single_document(file_path) for file_path in all_files]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\jose\privateGPT\ingest.py", line 68, in
return [load_single_document(file_path) for file_path in all_files]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\jose\privateGPT\ingest.py", line 56, in load_single_document
return loader.load()[0]
^^^^^^^^^^^^^
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain\document_loaders\pdf.py", line 206, in load
return list(self.lazy_load())
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain\document_loaders\pdf.py", line 213, in lazy_load
yield from self.parser.parse(blob)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain\document_loaders\base.py", line 87, in parse
return list(self.lazy_parse(blob))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain\document_loaders\parsers\pdf.py", line 35, in lazy_parse
text = extract_text(pdf_file_obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\high_level.py", line 175, in extract_text
interpreter.process_page(page)
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\pdfinterp.py", line 997, in process_page
self.render_contents(page.resources, page.contents, ctm=ctm)
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\pdfinterp.py", line 1016, in render_contents
self.execute(list_value(streams))
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\pdfinterp.py", line 1042, in execute
func(*args)
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\pdfinterp.py", line 902, in do_TJ
self.device.render_string(
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\pdfdevice.py", line 133, in render_string
textstate.linematrix = self.render_string_horizontal(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\pdfdevice.py", line 173, in render_string_horizontal
x += self.render_char(
^^^^^^^^^^^^^^^^^
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\converter.py", line 216, in render_char
textwidth = font.char_width(cid)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jose\AppData\Local\Programs\Python\Python311\Lib\site-packages\pdfminer\pdffont.py", line 921, in char_width
return cast(Dict[int, float], self.widths)[cid] * self.hscale
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
TypeError: can't multiply sequence by non-int of type 'float'

[JCMONFORT29]

Thank you for the reply, I experienced the exact same issue 3 days ago, I run the ingest on ~120 mb, let it run for 20+ hours on my M2 and eventually crashed ahah. I am trying right now the latest version of the code.