Build embeddings

[mishig25]

Add new command to doc-builder that creates embeddings from docs of huggingface libraries

Usage of the new command:

doc-builder embeddings [lib name] [docs path]
# example: doc-builder embeddings diffusers ~/diffusers/docs/source/en

How it works

1. Step 1: Create chunks from docs files

interface Chunk {
    text: string, // "# Effective and efficient diffusion\nGetting the `DiffusionPipeline` to generate images in a certain style or include what you want can be tricky ...
    source: string, // source url ex: hf.co/docs/transformers/model_doc/bert#transformers.BertConfig
    package_name: string, // ex: transformers, diffusers
}

doc-builder handles all the autodoc derivatives to crawl correct python objects and get their docstrings in markdowns. For example, find diffusers chunks at diffuers-chunks.json

2. Step 2: Embed those chunks using HF Inference Endpoints. In this case, We're using Snowflake/snowflake-arctic-embed-m

// same as Chunk & { embedding: number[] }
interface Embedding {
    text: string, // "# Effective and efficient diffusion\nGetting the `DiffusionPipeline` to generate images in a certain style or include what you want can be tricky ...
    source: string, // source url ex: hf.co/docs/transformers/model_doc/bert#transformers.BertConfig
    package_name: string, // ex: transformers, diffusers
    embedding: number[] // ex: [ 0.6372, -0.2380,  0.5643, -0.2773, -1.1663,  0.6670,  2.9874, -0.2245, -1.4203,  ...]
}

To save cost (for efficiency), I've turned on HF Inference Endpoints settings that makes it go to sleep if there was no usage in the last 15 mins. Therefore, I have this warmup check here:

doc-builder/src/doc_builder/build_embeddings.py

Lines 350 to 355 in 3e218b9

	# warm up API call
	output_warmup = query({"inputs": "Hello World!"})
	if isinstance(output_warmup, dict) and "error" in output_warmup:
	if output_warmup["error"] == "503 Service Unavailable":
	print("Waking up Embedding Inference Endpoints. Retrying in 5 minutes")
	time.sleep(300) # 5 minutes

3. Step 3: Upload the embeddings to vector database. We will be using Meilisearch Vector Store

Note: almost all changes of this PR is addition (i.e. it does not break any existing doc-builder functionalities, I will merge the PR once I test with meilisearch)

[julien-c]

would be more satisfying to receive an event when the endpoint is available (HTTP SSE events for instance), but i guess that's painful to do in Python

[mishig25]

does HF ednpoints provide events? cc: @philschmid

[Wauplin]

There is a huggingface_hub.InferenceEndpoint client in Python if you want but it does the same as you (wait for model to be available).

[Wauplin]

If instead of an API url you have the name of an inference endpoint, you can do:

from huggingface_hub import get_inference_endpoint

endpoint = get_inference_endpoint(name="...")
client = endpoint.wait().client
client.text_embedding(...)

[Wauplin]

@mishig25 Little update on inference endpoint. Before the .wait() call, you must do .resume(). If the endpoint scaled down to zero or have been paused, it will be restarted. So the snippet should look like this:

from huggingface_hub import get_inference_endpoint

endpoint = get_inference_endpoint(name="...")
client = endpoint.resume().wait().client
client.feature_extraction(...)

[julien-c]

ping again when you have this part i'd be interested in seeing it

[mishig25]

#506 (comment)

[Wauplin]

Very cool project! Looking forward to see it live 🎉

I've made a rough review of the current implementation. Feel free to ignore comments if you feel differently (especially if it's a POC for now)