Introduce a new index-level setting for disabling exact search based on the cardinality of the filtered documents#3109
Open
OrestisNer wants to merge 6 commits intoopensearch-project:mainfrom
Conversation
…t_search setting to disable exact search based on the cardinality of the filterIds Signed-off-by: Orestis Nerantzis <orestis.nerantzis@workday.com>
Signed-off-by: Orestis Nerantzis <orestis.nerantzis@workday.com>
OrestisNer
force-pushed
the
orestiner/disable-exact-search-threshold
branch
from
aa45784 to
ea5eb75
Compare
5 tasks
OrestisNer
changed the title
Signed-off-by: Orestis Nerantzis <orestis.nerantzis@workday.com>
Collaborator
|
@OrestisNer do you want to move this PR to in review state? |
Author
|
Hey @navneet1v, sorry for being late. Yes I will move it |
OrestisNer
requested review from
0ctopus13prime,
VijayanB,
Vikasht34,
heemin32,
jmazanec15,
junqiu-lei,
luyuncheng,
martin-gaievski,
naveentatikonda,
navneet1v,
ryanbogan,
shatejas and
vamshin
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introduce a new index-level setting called
index.knn.faiss.efficient_filter.disable_exact_search_thresholdthat allows users to disable the fallback to exact (brute-force) search, based on the cardinality of the filtered documents, which happens after an ANN search when using Faiss efficient filtering.Heavily inspired by this PR (which provides a setting for disabling exact search), with this new setting the goal is to provide more flexibility and allow users to optimize performance/relevance trade-offs based on their dataset sizes.
Related Issues
Resolves #2936
#3022 (comment)
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.