Why rocksdb? #149
Replies: 7 comments 1 reply
-
|
That's a great point! I started with RocksDB because at this time it was the only "good enough" key-value store with nice Rust bindings. I am currently considering moving from RocksDB to Sled that is much nicer to use and should get storage format stability soon. |
Beta Was this translation helpful? Give feedback.
-
|
Why not use TIKV? |
Beta Was this translation helpful? Give feedback.
-
|
TIKV is definitely an option. But it's a distributed store, so it would significantly change Oxigraph target from an easy to embed RDF database to a distributed RDF database. This would change a lot how Oxigraph is installed and run and the way Oxigraph stores data and optimize SPARQL queries (good tradeoffs are often not the same when data is distributed). It would be a great project but it's probably not Oxigraph (at least not Oxigraph with only me as main developer). I would love to see someone start such a project with TIKV. There would be a lot of room for code sharing between this possible project and Oxigraph (the Rio RDF parsers, the SPARQL parser and algebra...). |
Beta Was this translation helpful? Give feedback.
-
|
Is there a way to do an abstract key-value interface and do pluggable backends? (I had this in mind with my toy content-addressable store.) |
Beta Was this translation helpful? Give feedback.
-
|
It's definitely possible to have an abstract key-value store interface. However, outside of the basic operations (get, insert, delete, iterator) the different key value store have very different behaviors (synchronous vs asynchronous data access, local vs remote data in the case of a distributed store, write-only vs read-write vs no transaction...). So, to work properly Oxigraph would have to be able to adapt itself to all these cases. It's especially true for the local vs distributed distinction: distributed store like TIKV are useful to handle very large amount of data so being able to answer not trivial SPARQL queries the evaliuator should think a lot about data placement, how to evaluate the query... I am afraid doing that would lead to a system not able to properly handle only one or two backends that it has been optimized with, and have a "fake" support for the others which would "work" in theory but would be too buggy and/or slow to be usable in practice. It's why I think focusing on a single backend would be better for now, except if someone is willing to commit enough time to make others work properly. Having something working properly with Sled is already a huge endeavor for the time I am currently able to commit to Oxigraph. The current three very similar backends are already quite a burden development wise. |
Beta Was this translation helpful? Give feedback.
-
|
Fair enough. I adopted this measure because I started with Berkeley DB but then moved to LMDB (the thing that drives newer OpenLDAPs) to distance myself from Oracle, but wanted to keep the door open for products like LevelDB/RocksDB/Tkrzw. I figured I would always be riding on top of a direct-attached key-value store of some kind that was transaction-capable, and the layout itself within the key-value store would be identical (or near-identical) between implementations. So the abstraction wasn't for arbitrary databases, just the subset that met the desired criteria. |
Beta Was this translation helpful? Give feedback.
-
|
redb just released their 1.0 version - and they are trying to be a Rust-based rocksdb alternative. Has anyone evaluated it? |
Beta Was this translation helpful? Give feedback.
-
|
Not yet. It would be great to evaluate it indeed. |
Beta Was this translation helpful? Give feedback.
-
Rocksdb is a memory hog. Cdb and dgraph started with rocksdb but when they found memory and other issues they replaced rocksdb with their own implementation of key value store.
Beta Was this translation helpful? Give feedback.
All reactions