Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change borrow to move semantics for HnswIo index loading #19

Open
bwsw opened this issue Jun 10, 2024 · 1 comment
Open

Change borrow to move semantics for HnswIo index loading #19

bwsw opened this issue Jun 10, 2024 · 1 comment

Comments

@bwsw
Copy link
Contributor

bwsw commented Jun 10, 2024

Hello, now when you try to load the database from the disk, the code borrows structures from HnswIo. It produces a very uncomfortable use condition, either requiring a singleton or, as I recently did - a thread or async with a loop, so the borrow checker can ensure the correctness. Both variants are very constraining.

I propose changing this behavior to "move" behavior when Hnsw consumes HnswIo rather than borrows its properties. As a result, its use will be significantly simplified. However, I still do not understand the role of this borrowing design because, from my perspective, HnswIo is just a loader, not a holder of loaded data.

If particular data must be used by the Hnsw index, it needs to be moved there. We can use various approaches like:

enum Data {
   DataWithoutSideEffect(bla),
   DataWithSideEffect(blabla)
}
@jean-pierreBoth
Copy link
Owner

Yes the borrow is cumbersome, i battled with that.
The problem is Hnswio keeps mmap data and allows a partial memory load to access a point without loading the structure.

I did not find a better solution or give the Datamap responsability to the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants