fix(storage): preserve Parquet schema annotations - #20424
Open
youngsofun wants to merge 1 commit into
Open
Conversation
youngsofun
marked this pull request as draft
September 2, 2026 01:41
youngsofun
force-pushed
the
codex/fix-parquet-schema-roundtrip
branch
12 times, most recently
from
September 7, 2026 08:38
4f3b33f to
818dce2
Compare
youngsofun
force-pushed
the
codex/fix-parquet-schema-roundtrip
branch
from
September 7, 2026 08:46
818dce2 to
9d2193b
Compare
youngsofun
marked this pull request as ready for review
September 7, 2026 10:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
Parquet table plans serialized their inferred schema by printing and reparsing the Parquet message type. That round-trip could normalize a legacy Decimal annotation from
converted_type=DECIMAL, logical_type=Noneinto an explicit Decimal logical type. Row-group readers then rejected the original file because its schema no longer exactly matched the plan.This PR:
Mixed-version query nodes retain the existing Arrow-derived fallback when they cannot decode the other plan format. That fallback is best-effort and may normalize legacy annotations; subsequent schema mismatches now identify this condition and suggest checking query-node versions. Rolling-upgrade correctness for this case is outside this PR's scope.
Tests
Validated with focused catalog serialization tests, Parquet storage schema-difference tests, clippy for the affected crates, formatting, shell syntax, and diff checks. The schema-difference logic test reuses existing Parquet fixtures to cover file-only/inferred-only direction, complete short lists, and truncated wide-schema lists.
This regression does not keep an integration fixture. A tiny single-file stage read does not guarantee that the scan plan is serialized to another query node, so it can pass with the old implementation. Using several tiny files would still depend on scheduler placement rather than deterministically exercising the faulty round-trip. The direct schema-bytes round-trip and full
ParquetTableInfoserde tests instead reproduce the normalization at its source and assert thatlogical_type=Noneis preserved.Type of change
AI assistance
This change is