Add support for CrateDB to LangChain LLM framework #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amotl
force-pushed
the
cratedb
branch
4 times, most recently
from
f8d8d49
to
85bf8c7
Compare
amotl
force-pushed
the
cratedb
branch
3 times, most recently
from
9fd7cd7
to
c139332
Compare
1 task
amotl
force-pushed
the
cratedb
branch
2 times, most recently
from
b35e066
to
b4289e6
Compare
amotl
pushed a commit
that referenced
this pull request
Oct 11, 2023
Initial commit of rl_chain code
amotl
force-pushed
the
cratedb
branch
7 times, most recently
from
8a8fc4e
to
e3d07c4
Compare
amotl
commented
Oct 19, 2023
docs/docs/modules/data_connection/document_loaders/sqlalchemy.mdx
Outdated
Show resolved
Hide resolved
libs/langchain/tests/integration_tests/document_loaders/test_sqlalchemy_cratedb.py
Outdated
Show resolved
Hide resolved
amotl
force-pushed
the
cratedb
branch
5 times, most recently
from
29cf863
to
f75a3d7
Compare
It is a special adapter which provides similarity search across multiple collections. It can not be used for indexing documents.
The CrateDB adapter works a bit different compared to the pgvector adapter it is building upon: Because the dimensionality of the vector field needs to be specified at table creation time, but because it is also a runtime parameter in LangChain, the table creation needs to be delayed. In some cases, the tables do not exist yet, but this is only relevant for the case when the user requests to pre-delete the collection, using the `pre_delete_collection` argument. So, do the error handling only there instead, and _not_ on the generic data model utility functions.
…eddings The performance gains can be substantially.
The test cases can be written substantially more elegant.
The CrateDB SQLAlchemy dialect needs more love, so it was separated from the DBAPI HTTP driver.
kneth
reviewed
Oct 25, 2024
Comment on lines
52
to
60
| def embed_query(self, text: str) -> List[float]: | ||
| """Return consistent embeddings for the text, if seen before, or a constant | ||
| one if the text is unknown.""" | ||
| return self.embed_documents([text])[0] | ||
| if text not in self.known_texts: | ||
| return [float(1.0)] * (self.dimensionality - 1) + [float(0.0)] | ||
| return [float(1.0)] * (self.dimensionality - 1) + [ | ||
| float(self.known_texts.index(text)) | ||
| ] |
There was a problem hiding this comment.
Can we create an issue to track the work?
Those pages have been submitted to LangChain already.
At LangChain and CrateDB, we are maintaining a set of educational resources to learn a bit about their dynamic duo. That may spark your interest? |
amotl
commented
Oct 29, 2024
There was a problem hiding this comment.
This document is the main entrypoint that reflects details about CrateDB on the LangChain documentation, in its major LangChain Providers section. Please review it correspondingly, and suggest relevant edits. Thanks!
This was referenced Oct 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About
Discussing the patch to add support for CrateDB to LangChain, to be submitted upstream. Do not merge.
What's inside
FLOAT_VECTOR/KNN_MATCHfunctionality through LangChain's vector store subsystem.Documentation
Notebooks
Backlog
FIXMEandTODOremarks./cc @matriv, @seut, @marijaselakovic, @karynzv: You may also want to have a review on it? Thanks!