Skip to content

refactor(storage): refine recluster levels and log level IO - #20443

Open
zhyass wants to merge 2 commits into
databendlabs:mainfrom
zhyass:feat_hilbert_recluster
Open

refactor(storage): refine recluster levels and log level IO#20443
zhyass wants to merge 2 commits into
databendlabs:mainfrom
zhyass:feat_hilbert_recluster

Conversation

@zhyass

@zhyass zhyass commented Sep 6, 2026

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Restrict aggressive recluster mixing to levels 1–3; keep level 0 and each level >= 4 separate, with explicit group ordering before the task budget is consumed.
  • Determine promotion from the highest input level's share of uncompressed block bytes: at least 50% advances to max + 1; otherwise keep max. For equal-sized blocks, 1,1,3 -> 3, 1,3,3 -> 4, and 1,2,3 -> 3. Preserve the executor's base-level-plus-one convention.
  • Add structured fuse insert output, fuse recluster input, and fuse recluster output logs with table_id, level, block_count, block_size, and file_size. Share ClusterLevelLogStats; preserve per-block insert levels before segment reduction, and exclude unchanged remained blocks from recluster output.
  • Consolidate level regression cases and remove redundant tests.

These logs describe selected input and pre-commit write observations, not committed snapshot deltas or complete failed-operation IO accounting. Input levels use the current cluster key's effective statistics; output levels reflect written blocks, including perfect level -1 and NULL for missing cluster statistics. No per-round correlation or new history table is introduced.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

AI assistance

  • AI usage: An AI coding assistant helped draft and revise the recluster level calculation, diagnostic logs, regression tests, and PR summary. The responsible human reviewed the diff and directed the simplifications.
  • Responsible human: @zhyass
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

Mix only levels 1-3 in aggressive mode and promote based on the highest level's share of logical block bytes.

Share cluster-level log statistics for insert and recluster, exclude remained blocks from rewrite output, and trim redundant tests.
@github-actions github-actions Bot added the pr-bugfix this PR patches a bug in codebase label Sep 6, 2026
@zhyass zhyass changed the title fix(storage): refine recluster levels and log level IO feat(storage): refine recluster levels and log level IO Sep 6, 2026
@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Sep 6, 2026
@zhyass zhyass removed the pr-bugfix this PR patches a bug in codebase label Sep 6, 2026
@zhyass zhyass changed the title feat(storage): refine recluster levels and log level IO refactor(storage): refine recluster levels and log level IO Sep 6, 2026
@github-actions github-actions Bot added the pr-refactor this PR changes the code base without new features or bugfix label Sep 6, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca9df16418

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/storages/fuse/src/operations/common/meta/mutation_log.rs Outdated
Comment thread src/query/catalog/src/plan/partition.rs
@zhyass

zhyass commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 5073acad02

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

🤖 CI Job Analysis

Workflow: 34052607323

📊 Summary

  • Total Jobs: 32
  • Failed Jobs: 3
  • Retryable: 0
  • Code Issues: 3

NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

  • mac_check: Not retryable (Code/Test)
  • linux / check: Not retryable (Code/Test)
  • linux / test_unit: Not retryable (Code/Test)

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant