This repository has been archived by the owner on Oct 30, 2024. It is now read-only.
enhance: unpack archives before ingestion #100
Comments
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Currently, handling errors coming from ingesting documents within an archive can get pretty nasty.
Also, the way we do it right now cannot be parallelized for speedup and first collects all documents/chunks before doing the embedding. Missed opportunity for concurrency here and highly error-prone.
The text was updated successfully, but these errors were encountered: