Shell for mutable object

[streetycat]

#206

I will complete it in my project for Group.

[lurenpluto]

Currently inside cyfs-stack, StorageObject is used to store traditional data in the form of object, so in the new GC system, all StorageObjects will be treated as free roots.

However, if a StorageObject is used as a ShellObject, the GC mechanism needs to be improved in design, and it feels like it might be more friendly to introduce a new shell object?

[streetycat]

Maybe it's better to create a new type for clearer semantics, there are several reasons to use StorageObject:

• I don't known the background for StorageObject.
• It can support the function.
• It can be indexed by id with all storaged data when we need to retrieve the data in storage.

If it wants to introduce system-level customization, I think it's unnecessary. it's better to create a new object.

[streetycat]

By the way, what's mean with follow:

all StorageObjects will be treated as free roots.

It will be create/update/remove by manual? or timeout?

[lurenpluto]

Maybe it's better to create a new type for clearer semantics, there are several reasons to use StorageObject:

• I don't known the background for StorageObject.
• It can support the function.
• It can be indexed by id with all storaged data when we need to retrieve the data in storage.

If it wants to introduce system-level customization, I think it's unnecessary. it's better to create a new object.

There is nothing wrong with using StorageObject for ShellObject itself, except that StorageObject is now used as the default 'free root' for GC, which is an implicit setting and not quite correct in itself.

cyfs-stack internal 'free roots' using StorageObject are specified with a special string ID, so a list can be compiled and processed in GC

However, there is a risk that StorageObjects are used internally by cyfs-stack by default, so if an object with the same id is constructed externally, it will overwrite the existing objects inside cyfs-stack and lead to various unpredictable problems.
So one of the following permission design should be added:

1. StorageObject, only the same zone and system permissions can put_object, the rest of the put_object operations are rejected;
2. Key StorageObject inside the protocol stack, using high privileges(current device+system dec) at object layer system to avoid being overwritten by other dec apps

For the two options mentioned above, there are two different proposals that need to be studied further, as follows:

• If it is the first option, if you use this to do ShellObject, then the permissions part may need to be based on the type and ID to make a judgment, and it does not feel too friendly! So in this case, it feels more appropriate to add a new ShellObject, so that the StorageObject is only used to manage data inside the stack, requiring high privileges to operate

• But if it is the second option, then the StorageObject is available for external use, because the permission granularity is the object_id layer now, no need to differentiate by category

At present, I think the second solution is feasible, and we need to research it with the current implementation

[lurenpluto]

By the way, what's mean with follow:

all StorageObjects will be treated as free roots.

It will be create/update/remove by manual? or timeout?

It means "From the point of view of GC, these objects are always valid". The modification and management are used internally by the protocol stack. Generally, once created, they will not be deleted

[lurenpluto]

By the way, what's mean with follow:

all StorageObjects will be treated as free roots.

It will be create/update/remove by manual? or timeout?

It means "From the point of view of GC, these objects are always valid". The modification and management are used internally by the protocol stack. Generally, once created, they will not be deleted

Anyway, currently StorageObject is used inside the stack as a special object, used as a bridge to traditional data based object storage, so there is some special treatment inside this stack

So ShellObject can add a new and different object type if it can

[streetycat]

I see, but I think there needs to be more explicit information about this situation to avoid misunderstandings by other developers, for example:

1. define special section for similar objects.
2. make different flags.

[lurenpluto]

I see, but I think there needs to be more explicit information about this situation to avoid misunderstandings by other developers, for example:

1. define special section for similar objects.
2. make different flags.

Currently there is no clear specification for this piece of core objects, because how the kinds of objects are divided, also many different dimensions, may be a complex work, and later may need a special organization to be responsible for the division. But StorageObject this is really only cyfs-stack internal use, as a bridge between data and objects

[streetycat]

I have updated my sulution

Storage prototype

// ObjectShellObject
message ObjectShellDescContent {
    // true: storage `ObjectId` in `desc` only, without `ObjectDesc`;
    // false: encode the `ObjectDesc` in `desc`.
    bool is_object_id_only = 1;

    // true: calculate the `fix_content_hash` include `desc_sign`;
    // false: calculate the `fix_content_hash` without `desc_sign`.
    bool is_desc_sign_fix = 2;

    // true: calculate the `fix_content_hash` include `body_sign`;
    // false: calculate the `fix_content_hash` without `body_sign`.
    bool is_body_sign_fix = 3;

    // true: calculate the `fix_content_hash` include `nonce`;
    // false: calculate the `fix_content_hash` without `nonce`.
    bool is_nonce_fix = 4;

    // hash of fixed fields in `ObjectShellBodyContent`.
    // hash_buf = desc + body + [desc_signatures] + [body_signatures] + [nonce]
    // * any field with a value of `None` should be occupied with 1 byte with a value of 0.
    // fix_content_hash = sha256(hash_buf)
    bytes fix_content_hash = 5;
}

message ObjectShellBodyContent {
    // if is_object_id_only is true, `desc` is the encoded buffer of `ObjectId` of the original object.
    // otherwise, `desc` is the encoded buffer of the full `Desc` of the original object.
    bytes desc = 1;

    // `body` is the encoded buffer of the `Body` of the original object.
    optional bytes body = 2;

    // `desc_signatures` is the encoded buffer of the `Object.signs().desc_signs()` of the original object.
    optional bytes desc_signatures = 3;

    // `body_signatures` is the encoded buffer of the `Object.signs().body_signs()` of the original object.
    optional bytes body_signatures = 4;

    // `nonce` is the encoded buffer of the `nonce` of the original object.
    optional bytes nonce = 5;
}

A new object type

pub enum CoreObjectType {
    //...
    // A shell of an mutable `Object`, we can use it to storage `Object` with different `Body` and same `Desc` in `NOC`.
    ObjectShell = 42,

    // ...
}

Interface

#[derive(Clone)]
pub struct ObjectShell<O>
{
    desc: ShelledDesc<O::DescType>,
    body: Option<ObjectMutBody<O::ContentType, O>>,
    signs: ObjectSigns,
    nonce: Option<u128>,
    flags: ObjectShellFlags,
}

impl<O> ObjectShell<O>
{
    pub fn from_object<NO>(raw: &NO, flags: ObjectShellFlags) -> Self
    where
        NO: NamedObject<O> + RawEncode + for<'local> RawDecode<'local> + Clone,
    {
        Self {
            flags,
            desc: ShelledDesc::Desc(raw.desc().clone()),
            body: raw.body().clone(),
            signs: raw.signs().clone(),
            nonce: raw.nonce().clone(),
        }
    }

    pub fn shell_id(&self) -> ObjectId {
        self.to_storage().shell_id()
    }

    pub fn flags(&self) -> &ObjectShellFlags {
        &self.flags
    }

    pub fn with_full_desc(&self) -> bool {
        match self.desc {
            ShelledDesc::ObjectId(_) => false,
            ShelledDesc::Desc(_) => true,
        }
    }

    pub fn body(&self) -> &Option<ObjectMutBody<O::ContentType, O>> {
        &self.body
    }
    // update the raw object
    pub fn body_mut(&mut self) -> &mut Option<ObjectMutBody<O::ContentType, O>> {
        &mut self.body
    }

    pub fn signs(&self) -> &ObjectSigns {
        &self.signs
    }

    // update the signatures
    pub fn signs_mut(&mut self) -> &mut ObjectSigns {
        &mut self.signs
    }

    pub fn nonce(&self) -> &Option<u128> {
        &self.nonce
    }

    // update the nonce
    pub fn nonce_mut(&mut self) -> &mut Option<u128> {
        &mut self.nonce
    }

    pub fn try_into_object(
        mut self,
        desc: Option<&O::DescType>,
    ) -> BuckyResult<NamedObjectBase<O>> {
        // recover the original object from a shell.
    }

    fn from_storage(storage: &ObjectShellStorage) -> BuckyResult<Self> {
        // private interface: decode `ObjectShell` from the storaged object.
    }

    fn to_storage(&self) -> ObjectShellStorage {
        // private interface: encode `ObjectShell` to storaged object
    }
}

impl<O> RawEncode for ObjectShell<O>
{
    fn raw_measure(&self, purpose: &Option<cyfs_base::RawEncodePurpose>) -> BuckyResult<usize> {
        self.to_storage().raw_measure(purpose)
    }

    fn raw_encode<'a>(
        &self,
        buf: &'a mut [u8],
        purpose: &Option<cyfs_base::RawEncodePurpose>,
    ) -> BuckyResult<&'a mut [u8]> {
        self.to_storage().raw_encode(buf, purpose)
    }
}

impl<'de, O> RawDecode<'de> for ObjectShell<O>
{
    fn raw_decode(buf: &'de [u8]) -> BuckyResult<(Self, &'de [u8])> {
        let (storage, remain) = ObjectShellStorage::raw_decode(buf)?;
        Self::from_storage(&storage).map(|o| (o, remain))
    }
}

#[derive(Copy, Clone)]
pub struct ObjectShellFlags {
    is_object_id_only: bool,
    is_desc_sign_fix: bool,
    is_body_sign_fix: bool,
    is_nonce_fix: bool,
}

#[derive(Clone)]
enum ShelledDesc<D> {
    ObjectId(ObjectId),
    Desc(D),
}

// Private object for storage only.
// Anyone should always use the `ObjectShell` without the storage object directly.

#[derive(Debug, Clone, ProtobufEncode, ProtobufDecode, ProtobufTransform, Serialize)]
#[cyfs_protobuf_type(crate::codec::protos::ObjectShellDescContent)]
struct ObjectShellDescContent {
    is_object_id_only: bool,
    is_desc_sign_fix: bool,
    is_body_sign_fix: bool,
    is_nonce_fix: bool,
    fix_content_hash: HashValue,
}

impl DescContent for ObjectShellDescContent {
    fn obj_type() -> u16 {
        CoreObjectType::ObjectShell as u16
    }

    fn format(&self) -> u8 {
        OBJECT_CONTENT_CODEC_FORMAT_PROTOBUF
    }

    type OwnerType = SubDescNone; // no owner
    type AreaType = SubDescNone; // no area
    type AuthorType = SubDescNone; // no author
    type PublicKeyType = SubDescNone; // no public key
}

#[derive(Clone, Debug, ProtobufEncode, ProtobufDecode, ProtobufTransform)]
#[cyfs_protobuf_type(crate::codec::protos::ObjectShellBodyContent)]
struct ObjectShellBodyContent {
    desc: Vec<u8>,
    body: Option<Vec<u8>>,
    desc_signatures: Option<Vec<u8>>,
    body_signatures: Option<Vec<u8>>,
    nonce: Option<Vec<u8>>,
}

impl BodyContent for ObjectShellBodyContent {
    fn format(&self) -> u8 {
        OBJECT_CONTENT_CODEC_FORMAT_PROTOBUF
    }
}

impl ObjectShellBodyContent {
    fn hash(&self, flags: &ObjectShellFlags) -> HashValue {
        // hash of fixed fields in `ObjectShellBodyContent`.
        // hash_buf = desc + body + [desc_signatures] + [body_signatures] + [nonce]
        // * any field with a value of `None` should be occupied with 1 byte with a value of 0.
        // fix_content_hash = sha256(hash_buf)
    }
}

type ObjectShellType = NamedObjType<ObjectShellDescContent, ObjectShellBodyContent>;
type ObjectShellBuilder = NamedObjectBuilder<ObjectShellDescContent, ObjectShellBodyContent>;
type ObjectShellDesc = NamedObjectDesc<ObjectShellDescContent>;

type ObjectShellId = NamedObjectId<ObjectShellType>;
type ObjectShellStorage = NamedObjectBase<ObjectShellType>;