Index Anywhere #23
Index Anywhere #23
Conversation
Gozala
changed the title
|
|
||
| Prolly Trees provide a great foundation for efficient, potentially partial sync and were [demonstrated to be implementable on top of key value stores][prolly-tree kv]. We could implement Dynamo backend for [Okra] which would create means for replication and we could leverage some of the existing backends to replicate index in other nodes. | ||
|
|
||
| > ❗️ There is an assumption that Prolly Tree on top of Dynamo would be efficient enough to make added overhead non issue. |
There was a problem hiding this comment.
Even if that were the case, I do not want to depend on Dynamo.
There was a problem hiding this comment.
That is interesting, I was assuming that having prolly tree on top of dynamo would give best of the both worlds. Comparable performance in our current system and flexibility to choose backends optimized for constraints of their runtime.
I'd be very hesitant to switch to yet another DB in our system given our prior history those, or at least I would not tie two endeavors together
|
|
||
| > ℹ️ In fact records get grouped such that related records end up in the same group which could be leveraged by domain specific applications. | ||
|
|
||
| Put it more simply we could put datomic inspired datalog engine on top of Prolly Trees to regain desired flexibility and efficiency of queries and derive all other requirements from the Prolly Tree itself. As an added benefit Prolly Tree could be materialized an IPLD DAG making it ecosystem native. |
There was a problem hiding this comment.
Where would the Prolly Tree and its underlying data be stored? With the blob index?
If a user's data is replicated to multiple locations, is the prolly tree also replicated?
There was a problem hiding this comment.
I would imagine it would be stored in every user agent (that user has) probably as a local IPLD dag (although in some cases maybe indexdb or leveldb wolud be a better choice). On the server I imagine we would (at least initially) store it using key value store and dynamo would be what I'd try first.
For the reference canvas.xyz demonstrated an effective way to represent prolly tree using key value stores
📽️ Preview
This is a very early sketch of the idea I had been revealing in bits and pieces for a long time now. I hope to use this as space to iterate and evolve this into something concrete