[Core] Object store plugin #45
Conversation
|
Will try to take a look this week! |
There was a problem hiding this comment.
Sorry for the delay, and thanks for the proposal!
Overall I think the proposal looks reasonable. The main thing that is missing for me is a validating example. Ideally we would introduce one additional plugin to start, since this will let us validate the proposed API. Do you have one in mind that you're able to contribute to open source? Otherwise, we should work together to find something - off the top of my head:
- a Redis instance that's shared with non-Ray applications
- RDMA-based object store
It would also be good to see the CXL example fleshed out a bit more. One thing I didn't quite understand for example was how object fetching would work. Presumably we don't want to transfer the whole object via the normal raylet-raylet protocol anymore? But I wasn't sure how the current API proposal would be able to support that.
Also, if I understand correctly, this is only meant to use one plugin at a time, right?
One small note: could you create a new file with the proposal instead of overwriting README.md? Thanks!
README.md
Outdated
| virtual Status Authenticate(const std::string& user, const std::string& passwd) = 0; | ||
| virtual Status Authenticate(const std::string& secret) = 0; | ||
| /// The object storage optimized memory copy. | ||
| virtual void MemoryCopy(void* dest, const void* src, size_t len) = 0; |
There was a problem hiding this comment.
Can you say more about how this would get used?
There was a problem hiding this comment.
Currently memcpy() is used for data copy to object data buffer in writing object chunks (ObjectBufferPool::WriteChunk)
For objects with data buffer on CXL shared memory between hosts, additional optimization may be needed in addition to memcpy() to guarantee the cache coherence so that other hosts can directly access the shared object buffer directly. The plugin can provide that functionality.
README.md
Outdated
| New virtual functions of `ObjectStoreClientInterface` : | ||
| ```c++ | ||
| /// Authentication to the object store. | ||
| virtual Status Authenticate(const std::string& user, const std::string& passwd) = 0; |
There was a problem hiding this comment.
I think it would be best if this could be part of a more generic startup method, ideally one that already exists.
README.md
Outdated
|
|
||
| The plugin object store can be specified during Ray startup. The following CLI parameters, or `ray.init()` options are proposed for the plugin. | ||
| ```c++ | ||
| plugin_name: Optional[str] = ‘default’ |
There was a problem hiding this comment.
Probably best to make this part of the Ray system config, which can be overridden with RAY_<param> env variables.
Thank you for the review. It is meant that only one plugin is used. It is assumed that only one type of object store can be used. I'll create a new file as the base for further discussions. |
We propose object store plugin interfaces to allow Ray to support other object stores that may provide additional features without disruption of the overall ray object management.