support safe update #308
support safe update #308
Conversation
ShawnShawnYou
requested review from
inabao,
wxyucs,
LHT129 and
jiaweizone
as code owners
| LOG_ERROR_AND_RETURNS(ErrorType::INVALID_ARGUMENT, | ||
| fmt::format("failed to pretrain(invalid argument): base tag id " | ||
| "({}) doesn't belong to index", | ||
| base_tag_id)); |
There was a problem hiding this comment.
a minor fix: error message "bas tag id" to "base tag id"
| * @return result indicates whether the update operation is successful. | ||
| */ | ||
| virtual tl::expected<bool, Error> | ||
| UpdateVector(int64_t id, const DatasetPtr& new_base, bool need_fine_tune = false) { | ||
| UpdateVector(int64_t id, const DatasetPtr& new_base, bool force_update = false) { |
There was a problem hiding this comment.
p.s. We are not responsible for the consequences of this interface if the user specifies force_update = true
Codecov ReportAll modified and coverable lines are covered by tests ✅ |
| "ef_search": {} | ||
| }} | ||
| }})", | ||
| vsag::UPDATE_CHECK_SEARCH_L), |
| float neighbor_dist = | ||
| std::reinterpret_pointer_cast<hnswlib::HierarchicalNSW>(alg_hnsw_) | ||
| ->getDistanceByLabel(result.value()->GetIds()[i], new_base_vec); | ||
| if (neighbor_dist < self_dist) { |
There was a problem hiding this comment.
Here need set a range limit ?
There was a problem hiding this comment.
We can implicitly check the range by checking the neighborhood relationship, since it's hard to directly give a range that is general for every dataset. Besides, if the nearest neighbor of the updated vector is still the old vector, it remains our assumption of this interface (the updated vector stays in a nearby area around the original vector).
| for (int i = 0; i < result.value()->GetDim(); i++) { | ||
| float neighbor_dist = | ||
| std::reinterpret_pointer_cast<hnswlib::HierarchicalNSW>(alg_hnsw_) | ||
| ->getDistanceByLabel(result.value()->GetIds()[i], new_base_vec); |
There was a problem hiding this comment.
#309 provides a batch interface, which can be considered for use later on
There was a problem hiding this comment.
fix this in other pr?
Signed-off-by: zhongxiaoyao.zxy <[email protected]>
ShawnShawnYou
force-pushed
the
support-safe-update
branch
from
652697a to
9ef9613
Compare
ShawnShawnYou
added
the
kind/improvement
closes: #225