Feat: Make the encoding format forward and backward compatible

[Leeeon233]

Background

We may add other ContainerType in the future.
If developers need to ensure that all users have upgraded to the new version before they can enable the capabilities provided by future versions of Loro, this will be harmful to the developer experience.
The only thing holding back upgrades is the forward and backward compatibility of encoding formats.

Details

Loro has updates and snapshot two encoding formats. These two ways all need to encode arena, ops, and changes:

Arena

The arena is a better way to take advantage of columnar because there is a large amount of identical or incremental metadata. All arenas are in the last chunk in encoding format and encoded to&[u8]. Its compatibility depends on loro-dev/columnar#28.

Ops

Encoding and decoding ops are the most difficult part. We should parse ContainerType, ContainerID, and the Value/Prop of Op.

ContainerType

• add ContainerType::Unkown type with u8. We can infer the real ContainerType when the new version decodes an unknown type.

ContainerID & ContainerIdx

Container of type unknown will be registered in Loro arena, so they do have a corresponding ContainerIdx.
I use the first bit to infer the ContainerIdx whether is unknown, and 4 bits for ContainerType and 27 bits for index.

Value

• add an unknown ValueKind::Unknown(u8). For now, the unknown kind starts with 1 at the first bit, and 7 bits of the real ValueKind.
• add an enum FutureValue and an unknown FutureValue::Unkown{kind: u8, prop: i32, data: &[u8]}. Only we encode the future value, the length of bytes need to be encoded first. The newer value type will always be FutureValue.
• add InnerContent::Future(FutureInnerContent), and FutureInnerContent::Unknown{prop: i32, value: OwnedValue}. The content of unknown op will be degenerate to Unknown holding the decoded value. When encoding, we can process any content according to unified logic.

OwnedValue is the Value with ownership.

• any unknown bytes should not depend on arena.
• prop should not depend on arena.

[Leeeon233]

@zxch3n This is a proposal for the forward and backward compatibility of encoding formats. Can we add automated testing for this feature?

[zxch3n]

I'm not sure how to test it properly. Maybe we can use a new test-only cargo feature to simulate a new version of Loro and test it in another crate

[Leeeon233]

I'm not sure how to test it properly. Maybe we can use a new test-only cargo feature to simulate a new version of Loro and test it in another crate

I added Counter Container as Future Container Type and used it for fuzzing test.

It can cover:

• prop of Op
• decode unknown updates as InnerContent::Unknown{..}
• decode snapshot with unknown containers, using diff_calc to rebuild the state.
• FutureValue needs to encode bytes length first