-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot restore large index #66
Comments
hey @NicolasAlmerge , does it happen everytime u try to do so? |
Hey @JoanFM, yes it does. Small / medium files work well separately, but when trying to restore from a large file (~3GB, combining multiple small files), it crashes with this error. Maybe it is because of an array index out of bounds due to a 32-bit integer overflow, since the max 32-bit int is around 2.7B, but I do not know. |
can u provide some dummy example for me to reproduce? And what code exactly u are runnijg? together with jina and docarray versions? |
Hello,
I am using Python 3.10.9 and
vectordb==0.0.20
(latest as of this date), and I have a trouble when restoring saved data.I have two large files
A
andB
, and when I index them, snapshot them and restore them separately, everything works fine.When I read and parse files
A
andB
, index all the documents in both, then save them together, the snapshotting is successful. However, when trying to restore the data, I get the following error:Given previous tests I made and explanation, I suspect the issue is that the index is too large, hence raising the error. Does anyone know what can be done to fix this issue?
The text was updated successfully, but these errors were encountered: