Schema Evolution

[dominiklohmann]

Schemas change over time. Especially with schema inference, we often end up with multiple schemas of the same name that are actually different under the hood.

We have the schema id to filter this out right now, but that just shifts the burden onto the user. Instead, we want to transparently cast events on access of a partition to a superset schema for all schemas of the same name.

Definition of Done

Beta Give feedback

1. Agree on the approach
2. Implement the required changes

[dominiklohmann]

This was (indirectly) requested by a customer alongside #123—due to schema inference in the JSON parser, they have a lot of ever so slightly different schemas for Suricata, which on their end is configured not to write null values. This makes it impossible to merge partitions, thus slowing down exports from the node because of the suboptimal partition sizes.