Fix setting rescore as false in on_disk knn_vector query

[e-emoto]

Description

This change adds a flag to RescoreContext for tracking if rescore has been disabled by the KNN query parameters. This flag is set to not skip rescore by default, and is only set to skip rescore if "rescore": false is in the KNN query. UTs were added for the KNNQueryBuilderParser that covers the different cases for rescore. The below steps were also run with a debugging cluster to verify that the rescore was being run/skipped correctly based on the query parameters.

Manual Testing Steps

curl -X PUT "localhost:9200/test_index" -H 'Content-Type: application/json' -d'
{
  "settings" : {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "test_field1": {
        "type": "knn_vector",
        "dimension": 8,
        "space_type": "innerproduct",
        "data_type": "float",
        "mode": "on_disk",
        "compression_level": "16x"
      },"test_field2": {
        "type": "knn_vector",
        "dimension": 4,
        "space_type": "innerproduct",
        "data_type": "float",
        "mode": "on_disk",
        "compression_level": "16x"
      }
    }
  }
}
'

curl -X PUT "localhost:9200/_bulk" -H 'Content-Type: application/json' -d'
{ "index": { "_index": "test_index" } }
{ "test_field1": [1.5, 5.5, 3.5, 2.5, 5.5, 4.5, 7.5, 1.5], "test_field2": [1.5, 5.5, 3.5, 2.5]}
{ "index": { "_index": "test_index" } }
{ "test_field1": [2.5, 3.5, 7.5, 4.5, 1.5, 3.5, 5.5, 6.5], "test_field2": [2.5, 3.5, 7.5, 4.5]}
{ "index": { "_index": "test_index" } }
{ "test_field1": [4.5, 5.5, 1.5, 6.5, 3.5, 4.5, 2.5, 1.5], "test_field2": [4.5, 5.5, 1.5, 6.5]}
{ "index": { "_index": "test_index" } }
{ "test_field1": [1.5, 5.5, 2.5, 1.5, 3.5, 7.5, 5.5, 1.5], "test_field2": [1.5, 5.5, 2.5, 1.5]}
{ "index": { "_index": "test_index" } }
{ "test_field1": [1.5, 5.5, 4.5, 2.5, 3.5, 2.5, 6.5, 7.5], "test_field2": [1.5, 5.5, 4.5, 2.5]}
{ "index": { "_index": "test_index" } }
{ "test_field1": [2.5, 3.5, 5.5, 7.5, 1.5, 4.5, 3.5, 6.5], "test_field2": [2.5, 3.5, 5.5, 7.5]}
'

rescore skipped:

curl -X GET "localhost:9200/test_index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "knn": {
      "test_field1": {
        "vector": [ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0 ],
        "k": 3,
        "rescore": false
      }
    }
  }
}
'

rescore does happen:

curl -X GET "localhost:9200/test_index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "knn": {
      "test_field2": {
        "vector": [ 4.0, 3.0, 2.0, 1.0 ],
        "k": 3
      }
    }
  }
}
'

rescore does happen:

curl -X GET "localhost:9200/test_index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "knn": {
      "test_field1": {
        "vector": [ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0 ],
        "k": 3,
        "rescore": {
            "oversample_factor": 2.5
        }
      }
    }
  }
}
'

Related Issues

Resolves #2360

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[jmazanec15]

Thanks @e-emoto. Lets add a test somewhere in https://github.com/opensearch-project/k-NN/tree/main/src/test/java/org/opensearch/knn/integ that confirms that it actually does end up disabling re-scoring. Confirm that it fails before and works after.

These tests would provide a good template: https://github.com/opensearch-project/k-NN/blob/main/src/test/java/org/opensearch/knn/integ/ModeAndCompressionIT.java

[jmazanec15]

nit: Do we need to specify this?

Failing check - rerunning