feat(query): backport data lineage to v1.2.925

[youngsofun]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Backport the data-lineage feature series to release/v1.2.925 so it can be released as v1.2.925-patch-12.

This includes plan-level lineage extraction, history-backed persistence, lineage traversal and neighbor lookup, catalog/View field resolution, permanent-by-default retention, View reconciliation with REFRESH LINEAGE, dedicated lineage configuration, explicit materialized CTE extraction, system-source filtering, and the View-definition safety invariant.

Motivation

v1.2.925-patch-11 does not contain the data-lineage feature. Customers pinned to the v1.2.925 maintenance line need the same lineage interfaces and persistence behavior without upgrading to a later release train. This backport brings the complete reviewed lineage series to that maintenance branch while preserving release-specific APIs and excluding unrelated main-branch features.

Backported PRs

• feat(query): reject alter view definition changes #20202 feat(query): reject alter view definition changes
• feat(query): extract query lineage from plans #20201 feat(query): extract query lineage from plans
• feat(query): add history-based data lineage #20235 feat(query): add history-based data lineage
• refactor(query): reshape lineage history metadata #20289 refactor(query): reshape lineage history metadata
• refactor(query): clarify get lineage object output #20296 refactor(query): clarify get lineage object output
• feat(query): add direct lineage neighbor lookup #20300 feat(query): add direct lineage neighbor lookup
• feat(query): support catalog and view table fields #20304 feat(query): support catalog and view table fields
• feat(query): retain lineage history by default #20311 feat(query): retain lineage history by default
• feat(query): refresh lineage for existing views #20315 feat(query): refresh lineage for existing views
• feat(query): configure lineage independently #20322 feat(query): configure lineage independently
• feat(query): extract explicit materialized CTE lineage #20327 feat(query): extract explicit materialized CTE lineage
• fix(query): ignore system sources in lineage #20349 fix(query): ignore system sources in lineage

Reviewer guide

The first 12 commits preserve the source-PR order above. Review them against their linked source PRs, then review the release-only fixes separately:

• 7896fd3d13 — fix(query): adapt lineage backport to v1.2.925
• 36fe9dffbf — fix(query): correct expression scan column typo: behavior-neutral rename of a typo already present in release/v1.2.925, required by the current Typos check.
• 56c3a2be7c — test(query): remove stale lineage harness source: completes the #20315 test move by removing the obsolete history-test source path.
• cdfbfccb20 — fix(query): correct inverted index comment typo: comment-only correction of another typo already present in release/v1.2.925 and exposed by the current Typos check.

ca1a4991a3 changes only the Dev workflow base filter so this release-targeted PR can run full CI. It is temporary and will be removed after the run completes.

For the four cherry-picks that required manual conflict resolution, the focused review areas are:

• #20201: SQL lineage test helpers under src/query/sql/tests/it/framework and src/query/sql/tests/it/planner/lineage.rs. The production lineage implementation follows the source PR; only the release-local test harness location/API differs.
• #20235: interpreter_table_create.rs, table_function_factory.rs, and tests/logging/history_table/check_logs_table.sh. Verify that the existing temporary-table lifecycle, stream_backlog, and release history checks remain intact around the new lineage calls.
• #20315: the REFRESH LINEAGE AST/binder/plan/access/interpreter paths. Review this as an allowlist: only lineage wiring was added; no Materialized View variants or handlers from the source PR's newer main-branch context are present.
• #20322: src/query/sql/tests/it/planner/lineage.rs. Only the release-local init_testing_globals_with_config import path differs while capture moves to [lineage].

Conflict choices that retain existing release code or omit unrelated main-branch code do not produce a standalone Git diff. They remain in their source commits so the intermediate commits do not intentionally introduce broken release behavior.

Backport conflict resolution

• #20201: kept the v1.2.925 SQL test harness under tests/it/framework instead of importing the later main-branch move into test-support; ported only lineage-specific test helpers. The metadata column iterator API adjustment is in the final release compatibility commit.
• #20235: preserved the release branch's CREATE TABLE temporary-table lifecycle and existing stream_backlog registration; added only lineage target-ID propagation and get_lineage. Kept release history resource checks unchanged.
• #20315: added only REFRESH LINEAGE AST, binder, plan, access, and interpreter wiring. Materialized View parser/plan/access code present in the main-branch context was intentionally excluded.
• #20322: kept the release-local test initialization path while switching capture to the dedicated [lineage] configuration.
• Release compatibility is isolated in fix(query): adapt lineage backport to v1.2.925: CatalogType::Paimon handling is omitted because that variant is not present in this release, and columns_by_table_index() results are converted from the release Vec API into iterators.

Risk and compatibility

• This is a large feature backport spanning planner, service, configuration, history transforms, table functions, and integration tests.
• ALTER VIEW ... AS ... is rejected after this change. CREATE OR REPLACE VIEW remains the supported definition-replacement path. This prevents persisted lineage from becoming stale.
• [lineage] is the public configuration entry point. Older binaries ignore the TOML section during rolling upgrades; the new CLI option must not be passed to older binaries.
• The backport intentionally does not include unrelated Materialized View, Paimon, SQL test-framework, or metadata iterator refactors from main.
• Full CI is required before merge, especially the dedicated lineage, query, config, and history-table jobs.

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

Related Issues

Source PRs: #20202, #20201, #20235, #20289, #20296, #20300, #20304, #20311, #20315, #20322, #20327, #20349.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Unit Tests

• cargo test -p databend-common-sql --test it planner::lineage -- --nocapture (21 passed)
• cargo test -p databend-common-ast --test it test_statement -- --nocapture (2 passed)
• cargo test -p databend-common-config --lib (17 passed)
• cargo test -p databend-common-tracing --lib (13 passed)
• cargo check -p databend-common-sql --lib

Stateless Tests

• Dedicated lineage sqllogictest suite is included under tests/lineage and will run in CI.
• cargo fmt --all -- --check
• git diff --check
• bash -n tests/logging/history_table/check_logs_table.sh
• Shell syntax validation for all tests/lineage/*.sh scripts.

AI assistance

• AI usage: An AI coding agent identified the lineage PR series, performed the release backport, resolved release/main conflicts, added compatibility adjustments, ran local validation, and prepared the PR description; I reviewed the resulting branch and behavior.
• Responsible human: @youngsofun
• The responsible human has read every line of this diff and can explain each change

---

This change is 

[github-actions]

🤖 CI Job Analysis (Retry 4)

Workflow: 33379206719

📊 Summary

• Total Jobs: 88
• Failed Jobs: 4
• Retryable: 0
• Code Issues: 4

🚫 MAX RETRIES REACHED

Maximum retry count (3) has been reached. Manual intervention required.

View Workflow

🔍 Job Details

• ❌ linux / test_compat_java_client_cluster: Not retryable (Code/Test)
• ❌ linux / test_private_tasks: Not retryable (Code/Test)
• ❌ linux / test_stateful_cluster: Not retryable (Code/Test)
• ❌ linux / test_stateful_standalone: Not retryable (Code/Test)

---

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).