[IDEA] Guide/process to build embeddings elsewhere and sync that to the server.

[stevennt]

Describe the feature you'd like

Problem: Embedding on the server takes a lot of time, resource, and in many cases error prone, especially when GB-s of data need to be processed, and likely again and again.

If a script can be developed to allow users to run the embeddings elsewhere, then sync that up to the server / db, then it would be very convenient.

Thanks.

[debanjum]

Hey @stevennt, could you share a bit more about your use-case? What's the expected scale of usage?

• Khoj can be configured to compute embeddings using the HuggingFace's serverless inference API. For production use-case you can use their dedicated inference endpoints. Just set the inference_endpoint and api_key fields when you create a embeddings model on the Khoj admin panel
• Note: Khoj only recomputes embeddings of modified content, not all content sent to it. So unless there are GBs of new data, the embeddings to recompute should be smaller

[stevennt]

Hi @debanjum, my use-case is as follows: supposed you have a VIP user who you want to onboard. He has, say, 100GB of existing data in various places. Time is not a problem, he wants to start next week. But if he has to go through all the steps to index 100GB of data himself (the process which will certainly have issues here and there given that data size and data quality), then it will be a deal breaker.

So if you can have an onboarding script that takes care of all those workloads and let the user known when all is ready, then it will be a very convenient onboarding process.

This is true for organizations as well when we onboard for multiple users. Many of them may even share the same data set, so an assisted onboarding process can definitely help.

[debanjum]

Hi @stevennt, thanks for sharing the details of your use-case. I'm assuming you already have access to the data to be synced for this user? If so, the process would look like:

1. Request the user to create and share with you a (temporary) API key from their account on your hosted Khoj instance
2. Create a script to send the data in batches to the Khoj via API
   • You will need to call the PATCH /api/content API endpoint
   • You will need to call the API with Authorization: Bearar <user_api_token> request header
   • Ensure the data to be indexed as a multi-part/form
   • The PATCH /api/content API endpoint is used to incrementally index user data.
     See the autogenerated API docs and its usage in code for reference

Once the indexing is completed, the user can chat with their data without needing to run any data sync themselves.

Warning: Your Khoj API key is generally used to connect your client app to the Khoj server. It should not generally be shared as it gives full access to your Khoj account

I'll convert this issue into a discussion, as creating such a data sync script within the Khoj repository is not currently planned