How is document chunking done?

[thistleknot]

I've seen in text-generation-webui plugins options to set stride. Curious how document chunking is determined. I imagine it's a setting I can set, or you abstracted it away from the user and went with a pre prescribed method.

[debanjum]

Sorry missed replying to this earlier!

• For non hierarchical documents we use the recursive text chunking logic in Langchain
• For known hierarchical documents like markdown, org-mode. we have a custom tree-aware document chunking logic that tries to keep parent and sibling context for each node
• The doc chunking is only done if the file content cannot fit in the embedding models max context

You can look at text_to_entries.py, markdown_to_entries.py and docx_to_entries.py for the chunking logic