Document Information Extraction Strategies

[TomLucidor]

After reading this I do have some small thoughts https://github.com/echohive42/AI-reads-books-page-by-page

1. Summaries by the page seems to be useful for finding "memorable quotes" but probably not suitable for all books, most have too much filler per page and often times they re-iterate too much "common sense" to be useful, but sometimes if some pages are strong appendix it would be better to index them directly instead of summarizing
2. Chapter-wise indexing and summarizing would be easier to filter out the noise for most books, since a lot of chapters are about small pieces of knowledge wrapped under long narratives, and they can be treated as fables/parables with certain biases or generalizations for visual mnemonics
3. Book-wise indexing and summarizing would only be needed for the "best sellers" where it is the same content uttered across hundreds of pages, in that case [No comment]
4. Mindmaps are probably very useful in document searches (assuming vector databases are insufficient), and probably a worthy complementary tool to Socratic Dialogue Agents
5. This whole sentence-paragraph-chapter-book-collection hierarchies needs to be explored further in ways of summarizing/indexing, vectorizing, and graphing relations (connecting chapters of different genres, between adjacent rather than opposing POVs)

Other "knowledge extractors" (with non-MarkDown formats):

• Tables and triplets creation https://github.com/whyhow-ai/knowledge-table
• Q&A generation https://github.com/geeks-of-data/knowledge-gpt
• For agents, they also need to pull knowledge out https://github.com/huangwl18/language-planner