Embeddings with langchain fail: "rpc error: code = Unavailable desc = error reading from server: EOF"

[Connor2573]

Thanks so much for your help and this amazing software!

LocalAI version:
d65214a commit on 4/24/2024

Environment, CPU architecture, OS, and Version:
Linux server 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
Using the chroma.from_documents() function with either the LocalAIEmbeddings or OpenAIEmbeddings from langchain as the embedding function causes:
ERR Server error error="rpc error: code = Unavailable desc = error reading from server: EOF" ip=192.168.X.XXX latency=2.232229359s method=POST status=500 url=/embeddings

To Reproduce
Run local-ai compiled with CUBLAS

Expected behavior
I expected this to create a vector database and properly return the embeddings.

Logs
embedding_log.txt

Additional context
I have only managed to get the bert-embeddings backend to work properly, making me think I might have some sort of installation issue. The sentence-transformers backend produces incorrect embeddings for me and I get an error mentioned in another issue when I try to run a model in gguf format.
Additionally, sending a curl request or a basic request through OpenAI's python package to the server works perfectly fine. It is only when using specifically the from_documents method that it fails. I feel it has to have something to do with the two warning messages in the provided logs.

[localai-bot]

Hello Connor2573,

Thank you for reporting the issue and providing such detailed information. We appreciate your support for the LocalAI project.

Based on the details you provided, here's a possible interpretation of the problem:

• It seems that the server generates an "EOF" (End Of File) error while reading data, hence the "rpc error: code = Unavailable desc = error reading from server: EOF" message. The error might be originating from a specific part of the server-side code when handling the embedding requests from the from_documents() function in langchain.
• The fact that other functionalities of LocalAI are working fine, as confirmed by the successful curl requests and OpenAI python package interactions, suggests that this might be a targeted issue with the mentioned function.
• Given that you've mentioned only the bert-embeddings backend works correctly, it could indicate a problem with the implementations for other backends (sentence-transformers, OpenAIEmbeddings, LocalAIEmbeddings). However, this is just an assumption, and a thorough inspection of the code is necessary to confirm this hypothesis.

The next steps could be:

1. Upgrade LocalAI to the latest version, in case there's a patch or update that addresses this issue.
2. Check the server logs for any more detailed messages that could provide further clues on the root cause of the "EOF" error. Make sure to monitor the server status as well for any abnormal activity.
3. If possible, isolate the specific part of your code that's responsible for handling embedding requests, and go through it line by line to identify potential problems, especially focusing on the areas where the two warning messages mentioned in the logs occur.
4. If the issue persists, consider opening a pull request to the LocalAI repository with a potential fix or a more detailed analysis of the issue. This could be a great opportunity to contribute to the project and help the community!

Please ensure to use the issue tracker on Github to report any further findings or additional bugs. Let's collaborate to solve this issue and make LocalAI better! Don't hesitate to reach out if you need further help.

I'm a bot, an experiment of @mudler.

Best regards,
The LocalAI Team

(AI-generated response based on the given scenario, not a direct input from a team member of LocalAI)

[Connor2573]

Managed to get this working but reducing the length of the documents I was feeding to it

[Connor2573]

The issue seems to now be that the embeddings themselves are all basically the same. Even though when I run the same code using the same model directly it works out fine. I have done this texting with the aio docker image.