UserWarning: Relevance scores must be between 0 and 1

[amotl]

About

When running the test cases, there is a warning now. Most probably, it has been introduced by changing the style of the similarity search query through GH-15, which in turn changed the value range of the returned CrateDB-native _score values.

/path/to/langchain/libs/langchain/langchain/schema/vectorstore.py:313:
UserWarning: Relevance scores must be between 0 and 1, got 
[(Document(page_content='foo', metadata={'page': '0'}), 1.414213562373095), (Document(page_content='bar', metadata={'page': '1'}), 1.0606601717798212), (Document(page_content='baz', metadata={'page': '2'}), 0.8485281374238569)]

Evaluation

CrateDB's _score values are computed by CrateDB on behalf of different criteria of the input SQL query expression, execution plan, or actual execution. In this manner, they don't directly convey any useful information about the actual vector search similarity distance.

Suggestion

Use a corresponding function provided by CrateDB to compute the similarity distance independently of the CrateDB-native _score value.

• Vector Store: Provide similarity distance functions as scalar functions crate/crate#14801

/cc @ckurze, @seut, @matriv

[amotl]

This one might also be interesting, because it discusses potential similar [sic!] woes with other vector stores.

• When search_type="similarity_score_threshold, retriever returns negative scores langchain-ai/langchain#10864

At least, it tells us that not every store is getting it right from the very beginning, wrt. what LangChain or other applications might want or expect.

[amotl]

Hi. CrateDB 5.8.0 added a vector_similarity() function, which may be applicable to improve the situation here. Thank you so much!

• Vector Store: Provide distance functions as scalar functions  crate/crate#15835

[amotl]

We are addressing this leftover issue on the upstream patch now.

• community: CrateDB: Vector Store langchain-ai/langchain#27710 (comment)