The fastest and most efficient binary compression solution for Odin
- OdinLZ4Extension
Odin - Inspector and Serializer is one of the most popular development tools for Unity, its power and functionality is incredible, one of the main functions is de/serialization of data, it is extremely important function for most who use "Odin".
If you use cloud solutions in your projects and store some data in a serialized form, then imagine how much data space you can save with fast and efficient compression.
The best available algorithm was chosen as the compression engine, this is LZ4, it is extremely fast and efficient.
Combining these two engineering wonders we get extremely fast de/serialization and more free space for data.
So, Odin + LZ4 = PROFIT đź’Ş
The best realization of the LZ4 algorithm. This solution is provided by Milos Krajewski under the MIT license.
Original LZ4 has been written by Yann Collet and original 'C' sources can be found here
More info here
If you find this project useful, star it, I will be grateful.
Serialization object:
[Serializable]
public class Eexperimental
{
public DirectionalLight DirectionalLight;
public SpotLight SpotLight;
public PointLight PointLight;
}
Configs:
- Binary format
FAST
level of compression
Status | Milestone |
---|---|
🚀 | Multithread de/compressing and de/serializing |
âś… | De/Serialization performance tests with and without compression |
đź“Ś | Cut out the main parts of the LZ4 engine and compile a lightweight .dll |
Backend | .Net | Odin | Compatible |
---|---|---|---|
IL2CPP | 4.x | 3.x | âś… |
- Odin
- System libraries
The
OdinLZ4Extension
does not contain any source materials of theSirenix
, you can getOdin - Inspector and Serializer
on the AssetStore
At the moment
OdinLZ4Extension
only supports Odin - Inspector and Serializer
If you are only using the Odin Serializer then this extension will not be successfully imported into the Unity project. The reason for this is differentdll
s and namespaces. If you are using only Odin Serializer and would like to have this extension, write me and I will add this point to the roadmap.
-
Add the OdinLZ4Extension to your project (how to do it)
-
Include LZ4 necessary libraries (how to do it)
-
Plugin namespaces
using K4os.Compression.LZ4;
using OdinLZ4Extension;
Done, use de/serialization + LZ4 in one line. [Examples]
Several methods are provided for serialization with subsequent compression.
byte[] SerializeValue<T>(T value, DataFormat format, SerializationContext ctx = null, OdinLZ4Level level = OdinLZ4Level.FAST)
Fast serialization in binary format.
byte[] LazySerialization<T>(T value, OdinLZ4Level level = OdinLZ4Level.FAST)
byte[] SerializeValue<T>(T value, DataFormat format, out List<UnityEngine.Object> unityObjects, SerializationContext ctx = null, OdinLZ4Level level = OdinLZ4Level.FAST)
byte[] LazySerialization<T>(T value, out List<UnityEngine.Object> unityObjects, OdinLZ4Level level = OdinLZ4Level.FAST)
Several convenient methods are provided for deserializing compressed objects.
TResult DeserializeValue<TResult>(byte[] bytes, DataFormat format, DeserializationContext ctx = null)
TResult LazyDeserialization<TResult>(byte[] bytes)
Decompresses and deserializes a value of a given type from the given byte array in the given format, using the given list of Unity objects for external index reference resolution
TResult DeserializeValue<TResult>(byte[] bytes, DataFormat format, List<UnityEngine.Object> referencedUnityObjects, DeserializationContext ctx = null)
Fast data deserialization from binary format, using the given list of Unity objects for external index reference resolution
TResult LazyDeserialization<TResult>(byte[] bytes, List<UnityEngine.Object> referencedUnityObjects)
When you compress serialized data, FAST
compression is applied by default. You can choose the compression level for your needs.
Available compression levels:
FAST
(maximum compression speed)OPTIMAL
(sweet spot, fast and efficient compression)MAX
(slower than others but most efficient)
Code example:
OdinLZ4API.SerializeValue(SERIALIZABLE_VALUE, DataFormat.Binary, level: OdinLZ4Level.OPTIMAL);
OdinLZ4API.LazySerialization(SERIALIZABLE_VALUE, OdinLZ4Level.MAX);
Code examples:
[Serializable] public class SimpleData { public int a; public string someData; }
[Serializable] public class RefersData { public GameObject player1, player2; public Vector3 somePos; }
[SerializeField] private SimpleData simpleData = new SimpleData() { a = 100, someData = "It's work!" };
[SerializeField] private RefersData refersData;
SimpleData deserializedSimpleData;
RefersData deserializedRefersData;
byte[] smplSerialization, refersSerialization;
List<UnityEngine.Object> refs;
// --- Serialization ---
// Base serialization
smplSerialization = OdinLZ4API.SerializeValue(simpleData, DataFormat.Binary, level:OdinLZ4Level.MAX);
// Lazy Serialization
smplSerialization = OdinLZ4API.LazySerialization(simpleData);
// Serialization with Unity object references
refersSerialization = OdinLZ4API.SerializeValue(refersData, DataFormat.Binary, out refs);
// Lazy Serialization with Unity object references
refersSerialization = OdinLZ4API.LazySerialization(refersData, out refs);
// --- Deserialization ---
// Base deserialization
deserializedSimpleData = OdinLZ4API.DeserializeValue<SimpleData>(smplSerialization, DataFormat.Binary);
// Lazy deserialization
deserializedSimpleData = OdinLZ4API.LazyDeserialization<SimpleData>(smplSerialization);
// Deserialization using the given list of Unity objects for external index reference resolution
deserializedRefersData = OdinLZ4API.DeserializeValue<RefersData>(refersSerialization, DataFormat.Binary, refs);
// Lazy deserialization using the given list of Unity objects for external index reference resolution
deserializedRefersData = OdinLZ4API.LazyDeserialization<RefersData>(refersSerialization, refs);
The package is available on the openupm registry. It's recommended to install it via openupm-cli.
openupm add com.inc8877.odinlz4extension
Open Packages/manifest.json
with your favorite text editor. Add the following line to the dependencies block.
{
"dependencies": {
"com.inc8877.odinlz4extension": "https://github.com/inc8877/OdinLZ4Extension.git",
}
}
You can easily upgrade data de/serialization that is already implemented in your project to de/serialization with de/compression.
To do this, it is enough to plugin the necessary namespaces and simply replace SerializationUtility
class to OdinLZ4API
.
Code example:
// Necessary namespaces
using K4os.Compression.LZ4;
using OdinLZ4Extension;
// Existing code
SerializationUtility.SerializeValue(SERIALIZABLE_VALUE, DataFormat.Binary);
SerializationUtility.DeserializeValue<SimpleData>(BYTES_ARRAY, DataFormat.Binary);
// Upgraded code
OdinLZ4API.SerializeValue(SERIALIZABLE_VALUE, DataFormat.Binary);
OdinLZ4API.DeserializeValue<SimpleData>(BYTES_ARRAY, DataFormat.Binary);
Below is a list of methods in the SerializationUtility
class that can be replaced by methods with compression:
// --- Serialization ---
byte[] SerializeValue<T>(T value, DataFormat format, SerializationContext context = null);
byte[] SerializeValue<T>(T value, DataFormat format, out List<UnityEngine.Object> unityObjects, SerializationContext context = null);
// --- Deserialization ---
T DeserializeValue<T>(byte[] bytes, DataFormat format, DeserializationContext context = null);
T DeserializeValue<T>(byte[] bytes, DataFormat format, List<UnityEngine.Object> referencedUnityObjects, DeserializationContext context = null);
Unity does not provide the necessary libraries for the LZ4, so you need to install them manually.
The libraries can be downloaded via NuGet, but since the Unity does not work well with it, you need to get the .dll
from the downloaded packages and put them into the Assets
folder.
If you don't want to mess with it, you can get the DLLs
in a .zip
file located in the OdinLZ4Extension root folder (path: Packages/OdinLZ4Extension/RelatedMaterials/SystemDLLs.zip
). Just unzip and place them in the project. (A good location for .dll
in the project would be the Assets/Plugins/System
folder)
At the moment the project has just begun its life and requires a lot of work, if you are reading this means you may have liked this project and you can help by doing the following:
- do performance tests (compare data serialization with and without compression and make a graph)
- expand functionality:
- serialization with compressing via multithreading
- a pool of serialization tasks with callbacks