Skip to content

fix(query): avoid rewrites for Parquet-compatible decimal widening - #20418

Draft
youngsofun wants to merge 1 commit into
databendlabs:mainfrom
youngsofun:codex/metadata-only-decimal-widen
Draft

fix(query): avoid rewrites for Parquet-compatible decimal widening#20418
youngsofun wants to merge 1 commit into
databendlabs:mainfrom
youngsofun:codex/metadata-only-decimal-widen

Conversation

@youngsofun

@youngsofun youngsofun commented Aug 31, 2026

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

ALTER TABLE ... MODIFY COLUMN currently rewrites the full table whenever a Decimal type changes, even when only the precision is widened and the persisted Parquet representation is unchanged. On large tables this turns a lossless schema change into a full scan and rewrite.

This PR makes eligible Decimal precision widening metadata-only when:

  • scale is unchanged;
  • precision only increases;
  • the Decimal runtime kind is unchanged;
  • the Parquet physical type is unchanged; and
  • FIXED_LEN_BYTE_ARRAY uses the same byte width.

Changes that cross a Decimal kind or Parquet-width boundary, change scale, or narrow precision continue to rewrite data.

Schema Decimal variants are not stable across meta protobuf round trips, so the decision derives Decimal width from precision rather than enum discriminants. Persisted statistics also keep their original DecimalSize; schema-aware ColumnStatistics and ClusterStatistics views temporarily retag compatible bounds and fail open for incompatible statistics.

Reducers, range pruning, TopN, REPLACE, partition Top-K, runtime scan ordering/filtering, and reclustering now consume the safe statistics abstraction. Compatible Decimal TopN values merge at the widest precision, while incompatible statistics are discarded without blocking writes. Missing statistics remain unknown instead of being represented by fabricated bounds.

The persisted statistics layout is unchanged. Existing segment and snapshot metadata remain readable, and metadata-only DDL preserves existing block locations.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Validated with:

  • cargo test -p databend-storages-common-table-meta --lib
  • cargo test -p databend-common-storages-fuse --lib cluster_statistics::tests
  • cargo test -p databend-query --test it parquet_rs::prune_pages -- --nocapture
  • cargo test -p databend-query --test it parquet_rs::prune_row_groups -- --nocapture
  • cargo clippy -p databend-storages-common-table-meta -- -D warnings
  • cargo fmt --all -- --check

The logic test covers metadata-only ALTER, predicate reads, post-ALTER writes, TopN, REPLACE, ANALYZE, COMPACT, reclustering, time travel, Decimal physical-width boundaries, and preservation of existing block locations.

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

AI assistance

  • AI usage: An AI coding agent assisted with root-cause analysis, storage and statistics implementation, regression tests, local review, and PR drafting
  • Responsible human: @youngsofun
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

@github-actions github-actions Bot added the pr-bugfix this PR patches a bug in codebase label Aug 31, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cdc2c24bde

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/interpreters/interpreter_table_modify_column.rs Outdated
@youngsofun
youngsofun force-pushed the codex/metadata-only-decimal-widen branch 2 times, most recently from 4a72d5b to 024cb69 Compare September 1, 2026 03:49
@youngsofun youngsofun changed the title fix(query): avoid rewriting data for decimal precision widening fix(query): avoid rewrites for Parquet-compatible decimal widening Sep 1, 2026
@youngsofun
youngsofun marked this pull request as draft September 1, 2026 07:44
@youngsofun
youngsofun force-pushed the codex/metadata-only-decimal-widen branch 2 times, most recently from 2258cad to 25be845 Compare September 2, 2026 14:33
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🤖 CI Job Analysis

Workflow: 34074383775

⛔️ CANCELLED

Higher priority request detected - retry cancelled to avoid conflicts.

View Workflow

@youngsofun
youngsofun force-pushed the codex/metadata-only-decimal-widen branch 2 times, most recently from f5774f5 to 6407a05 Compare September 2, 2026 18:12
@youngsofun
youngsofun force-pushed the codex/metadata-only-decimal-widen branch from 6407a05 to 801b58f Compare September 7, 2026 00:41
# Conflicts:
#	src/query/service/src/physical_plans/physical_top_n.rs
#	src/query/storages/fuse/src/operations/replace_into/mutator/replace_into_operation_agg.rs
@youngsofun
youngsofun force-pushed the codex/metadata-only-decimal-widen branch 3 times, most recently from 94bd908 to dcce5b2 Compare September 7, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix this PR patches a bug in codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants