Potential Features/ Nice to haves

[shahrukhx01]

@lalitpagaria @GirishPatel Here are some of my thoughts going forward,

1. Text summarization: To add text summarizer node which summarize long texts.
2. Knowledge Graph: To enable automated knowledge graph based on rules provided by user after sink and enable the user to perform analytics and search on knowledge graph
3. Full-Text Neural Search: Enable full-text neural search on raw data enables the user to search for any source of information.
4. Unsupervised Clustering: Use an unsupervised algorithm to group similar documents using clustering. Clustering can easily be done on embedding provided by for instance sentence transformers.
5. Topic models: To determine topics out of texts to automatically tag documents to different categories using topic modeling techniques.
6. Aggregate multiple news sources: This article has the best 10 news APIs, https://rapidapi.com/blog/rapidapi-featured-news-apis/ We can use, for instance, it pulls news from 10k data sources and allows 500 free requests/day: https://usearch.com/news-api We can aggregate the above api with the existing gnews api and have large enough news set. For tech news: https://github.com/HackerNews/API
7. User enters prompt/title for the post. GPT2/BERT based model generates text for user. User make appropriate changes and posts.
   IDEA taken from: https://contentyze.com/
8. Explore vanilla KGs from here: https://github.com/pykeen/pykeen

[lalitpagaria]

Thanks @shahrukhx01 for suggestions these are relevant.
We had Text summarization, Text Generator, and Neural Search in original future concept image, but later we removed them.
Once we able to integrate Haystack with Obsei then we would able to leverage all 1-3 and multi mode search. Also we have idea to use Haystack as sink as well, so these test information can be pushed to Haystack to better searching. It is really go for workflow like new article, confluence, blogs, FAQ and image search.

[lalitpagaria]

Yes true

[lalitpagaria]

For KG this looks promising https://github.com/pykeen/pykeen

[shahrukhx01]

nice, will check it out!

[cthoyt]

FYI most of the knowledge graphs in PyKEEN are pre-stratified into training/testing/validation splits. However, you can trivially concatenate them. I'm not sure exactly what your use cases are, but many of the benchmark graphs in PyKEEN are pre-processed from larger, more complete KGs (e.g., FB15k is a subset of freebase)

[lalitpagaria]

@cthoyt Yes you are correct. Use case is to understand unstructured text continuously and which in case of in prediction phase. So PyKEEN doesn't fit in this use case.

[shahrukhx01]

@lalitpagaria I'll keep updating the above post, to keep track of everything.