diff --git a/docs/cn/guides/20-self-hosted/04-references/node-config/query-config.md b/docs/cn/guides/20-self-hosted/04-references/node-config/query-config.md index ac531f1d71..f888da1f4e 100644 --- a/docs/cn/guides/20-self-hosted/04-references/node-config/query-config.md +++ b/docs/cn/guides/20-self-hosted/04-references/node-config/query-config.md @@ -6,7 +6,7 @@ import FunctionDescription from '@site/src/components/FunctionDescription'; import LanguageDocs from '@site/src/components/LanguageDocs'; import DetailsWrap from '@site/src/components/DetailsWrap'; - + 本页介绍 [databend-query.toml](https://github.com/databendlabs/databend/blob/main/scripts/distribution/configs/databend-query.toml) 配置文件中可用的 Query 节点配置。 @@ -77,6 +77,25 @@ on = true | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | on | 是否在 Databend Query 节点上启用私有化 Task 调度与执行，默认为 `false`。 | +## [lineage] 部分 + +使用 `[lineage]` 配置捕获并持久化对象级和列级数据血缘： + +```toml +[lineage] +on = true +# retention = 720 +``` + +请在所有 Query 节点上使用相同配置，并在修改配置后重启节点。启用数据血缘后，Databend 会自动配置内部历史存储；不要将 `lineage_history` 添加到 `[log.history.tables]`。 + +| 参数 | 描述 | +|------|------| +| on | 是否启用血缘捕获和持久化，默认为 `false`。 | +| retention | 可选的 DML 血缘保留时长，单位为小时。省略时永久保留血缘。 | + +有关使用方法，请参见[数据血缘](/guides/data-management/data-lineage)。 + ## [log] 部分 该部分可包含以下子部分：[log.file]、[log.stderr]、[log.query] 和 [log.tracing]。 diff --git a/docs/cn/guides/57-data-management/05-data-lineage.md b/docs/cn/guides/57-data-management/05-data-lineage.md new file mode 100644 index 0000000000..cebf9469fc --- /dev/null +++ b/docs/cn/guides/57-data-management/05-data-lineage.md @@ -0,0 +1,130 @@ +--- +title: 数据血缘 +--- + +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +数据血缘用于展示数据如何从源对象流向目标对象。你可以使用数据血缘了解对象依赖关系、评估变更影响、排查数据管道问题，以及将派生列追溯到其源列。 + +Databend 会记录对象级和列级关系： + +- **上游血缘（Upstream Lineage）**：标识为当前对象提供数据的表、视图或 Stage。 +- **下游血缘（Downstream Lineage）**：标识使用当前对象数据的其他对象。 +- **列级血缘（Column Lineage）**：展示源列到派生目标列的映射关系。 + +![Databend Cloud 中的表级和列级血缘](/img/guides/data-lineage.png) + +## 启用数据血缘 + +对于提供 **Lineage** 页签的 Databend Cloud Warehouse，血缘配置由服务管理。私有化部署需要在每个 Query 节点的 `databend-query.toml` 中添加以下配置，然后重启节点： + +```toml title="databend-query.toml" +[lineage] +on = true +``` + +默认情况下，血缘历史会永久保留。如需设置固定的保留时长，可通过 `retention` 指定小时数，例如： + +```toml title="databend-query.toml" +[lineage] +on = true +retention = 720 +``` + +请仅使用专用的 `[lineage]` 配置，不要将内部表 `lineage_history` 添加到 `[log.history.tables]`。完整配置说明请参见 [Query 节点配置](/guides/self-hosted/references/node-config/query-config#lineage-部分)。 + +## 生成血缘关系 + +启用数据血缘后，Databend 会自动记录由 `CREATE TABLE ... AS SELECT`、`CREATE VIEW`、`INSERT ... SELECT`、多表 `INSERT`、`REPLACE`、`MERGE` 和 `COPY` 等操作产生的关系。通过 Stream 读取数据时，血缘会解析到其底层表。 + +以下示例创建了一条包含两跳关系的血缘链路： + +```sql +CREATE OR REPLACE DATABASE lineage_demo; + +CREATE OR REPLACE TABLE lineage_demo.fact_orders ( + order_id BIGINT, + customer_id BIGINT, + amount DECIMAL(12, 2), + order_time TIMESTAMP +); + +CREATE OR REPLACE TABLE lineage_demo.agg_customer_sales AS +SELECT + customer_id, + sum(amount) AS total_amount, + count(*) AS order_count, + max(order_time) AS last_order_time +FROM lineage_demo.fact_orders +GROUP BY customer_id; + +CREATE OR REPLACE TABLE lineage_demo.customer_segments AS +SELECT + customer_id, + total_amount, + order_count, + if(total_amount >= 1000, 'high_value', 'standard') AS segment, + now() AS updated_at +FROM lineage_demo.agg_customer_sales; +``` + +## 查看血缘关系 + +在 Databend Cloud 中，通过 Database Explorer 打开一个表或视图，然后选择 **Lineage** 页签。血缘图会显示上下游对象；如果存在列级血缘，还会显示列之间的连接关系。 + +如需通过 SQL 查询血缘，请使用 [`GET_LINEAGE`](/sql/sql-functions/table-functions/get-lineage) 表函数： + +```sql +SELECT + distance, + source_object_database, + source_object_name, + target_object_database, + target_object_name +FROM GET_LINEAGE( + 'lineage_demo.agg_customer_sales', + 'TABLE', + 'UPSTREAM', + 2 +) +ORDER BY distance; +``` + +查询列级血缘时，请使用带限定符的列名，并将对象域指定为 `COLUMN`： + +```sql +SELECT + distance, + source_object_name, + source_column_name, + target_object_name, + target_column_name +FROM GET_LINEAGE( + 'lineage_demo.customer_segments.segment', + 'COLUMN', + 'UPSTREAM', + 2 +) +ORDER BY distance; +``` + +## 刷新现有视图的血缘 + +启用数据血缘后创建的视图会被自动追踪。如果部署中已存在视图，请先预览缺失或过期的关系，然后再进行刷新： + +```sql +REFRESH LINEAGE FOR ALL VIEWS DRY RUN; +REFRESH LINEAGE FOR ALL VIEWS; +``` + +该命令会校准 `default` Catalog 中所有视图的血缘关系。结果只显示需要变更或无法处理的视图，不显示未发生变化的视图。执行命令需要全局 `SUPER` 权限。有关输出列的详细说明，请参见 [`REFRESH LINEAGE`](/sql/sql-commands/ddl/view/refresh-lineage)。 + +## 使用限制 + +- `GET_LINEAGE` 最多可遍历五跳关系。 +- `system` 和 `information_schema` 中的对象不会作为血缘源记录。 +- Stage 支持对象级血缘，但 Stage 文件字段无法提供稳定的列级映射。 +- 外部 Catalog 对象可作为血缘端点显示，但不会继续跨越外部 Catalog 边界遍历。 +- 查询结果仅包含当前角色可见的对象。 diff --git a/docs/cn/guides/57-data-management/index.md b/docs/cn/guides/57-data-management/index.md index c7118a454f..ec2bc7c12d 100644 --- a/docs/cn/guides/57-data-management/index.md +++ b/docs/cn/guides/57-data-management/index.md @@ -9,4 +9,5 @@ title: 数据管理 | **[数据生命周期](./01-data-lifecycle.md)** | 创建和管理对象 | • 数据库和表
• 外部表
• Stream 和视图
• 索引和 Stage | • CREATE/DROP/ALTER
• SHOW TABLES
• DESCRIBE TABLE | | **[数据恢复](./02-data-recovery.md)** | 访问和恢复历史数据 | • 时间回溯
• 闪回表
• 备份和恢复
• AT 和 UNDROP | • SELECT ... AT
• FLASHBACK TABLE
• BENDSAVE BACKUP | | **[数据保护](./03-data-protection.md)** | 安全访问和防止数据丢失 | • 网络策略
• 访问控制
• 时间回溯和故障安全
• 数据加密 | • NETWORK POLICY
• GRANT/REVOKE
• USER/ROLE | -| **[数据回收](./04-data-recycle.md)** | 释放存储空间 | • VACUUM 命令
• 保留策略
• 孤立文件清理
• 临时文件管理 | • VACUUM TABLE
• VACUUM DROP TABLE
• DATA_RETENTION_TIME | \ No newline at end of file +| **[数据回收](./04-data-recycle.md)** | 释放存储空间 | • VACUUM 命令
• 保留策略
• 孤立文件清理
• 临时文件管理 | • VACUUM TABLE
• VACUUM DROP TABLE
• DATA_RETENTION_TIME | +| **[数据血缘](./05-data-lineage.md)** | 追踪数据流和依赖关系 | • 上游和下游
• 对象级和列级血缘
• 影响分析 | • GET_LINEAGE
• REFRESH LINEAGE
• 血缘图 | diff --git a/docs/cn/sql-reference/10-sql-commands/00-ddl/05-view/index.md b/docs/cn/sql-reference/10-sql-commands/00-ddl/05-view/index.md index ed15ae928c..9bb54300c3 100644 --- a/docs/cn/sql-reference/10-sql-commands/00-ddl/05-view/index.md +++ b/docs/cn/sql-reference/10-sql-commands/00-ddl/05-view/index.md @@ -12,6 +12,7 @@ title: 视图（View） | [ALTER VIEW](ddl-alter-view.md) | 为现有视图分配或移除 Tag | | [DROP VIEW](ddl-drop-view.md) | 删除视图 | | [物化视图](materialized-view.md) | 创建并维护由物理存储支持的物化视图 | +| [REFRESH LINEAGE](refresh-lineage.md) | 回填或校准现有视图的血缘关系 | ## 视图信息 diff --git a/docs/cn/sql-reference/10-sql-commands/00-ddl/05-view/refresh-lineage.md b/docs/cn/sql-reference/10-sql-commands/00-ddl/05-view/refresh-lineage.md new file mode 100644 index 0000000000..49db291993 --- /dev/null +++ b/docs/cn/sql-reference/10-sql-commands/00-ddl/05-view/refresh-lineage.md @@ -0,0 +1,71 @@ +--- +title: REFRESH LINEAGE +sidebar_position: 7 +--- + +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +回填或校准 `default` Catalog 中现有视图的血缘关系。在已有视图的部署中启用数据血缘后，请使用此命令进行回填。启用数据血缘后创建的视图会被自动追踪。 + +执行此命令需要全局 `SUPER` 权限，并且必须已启用数据血缘。请参见[数据血缘](/guides/data-management/data-lineage#启用数据血缘)。 + +## 语法 + +```sql +REFRESH LINEAGE FOR ALL VIEWS [ DRY RUN ] +``` + +`DRY RUN` 只计算并报告变更，不会写入数据。建议先使用该选项检查刷新将执行的操作。 + +## 输出列 + +| 列 | 描述 | +|----|------| +| `object_domain` | 对象域，当前为 `VIEW`。 | +| `catalog` | 视图所属的 Catalog，当前为 `default`。 | +| `database` | 视图所属的数据库。 | +| `object_name` | 视图名称。 | +| `status` | `DRY_RUN`、`REFRESHED` 或 `ERROR`。 | +| `edge_count` | 当前视图定义中发现的血缘边数量。 | +| `upsert_count` | 需要新增或更新的缺失或已变更血缘边数量。 | +| `delete_count` | 需要删除的过期血缘边数量。 | +| `error` | `status` 为 `ERROR` 时的错误信息；其他情况为 `NULL`。 | + +没有变更且处理成功的视图不会出现在结果中。 + +## 示例 + +预览现有视图需要执行的变更： + +```sql +REFRESH LINEAGE FOR ALL VIEWS DRY RUN; +``` + +应用变更： + +```sql +REFRESH LINEAGE FOR ALL VIEWS; +``` + +命令完成后，可通过 [`GET_LINEAGE`](/sql/sql-functions/table-functions/get-lineage) 查询视图的上游血缘： + +```sql +SELECT + distance, + source_object_database, + source_object_name, + target_object_database, + target_object_name +FROM GET_LINEAGE( + 'lineage_demo.sales_view', + 'VIEW', + 'UPSTREAM', + 1 +); +``` + +:::note +如需修改逻辑视图定义，请使用 [`CREATE OR REPLACE VIEW`](ddl-create-view.md)。不支持 `ALTER VIEW ... AS ...`，因为不重新创建视图就修改定义可能导致已持久化的血缘关系不一致。 +::: diff --git a/docs/cn/sql-reference/20-sql-functions/17-table-functions/get-lineage.md b/docs/cn/sql-reference/20-sql-functions/17-table-functions/get-lineage.md new file mode 100644 index 0000000000..fd7fd9a486 --- /dev/null +++ b/docs/cn/sql-reference/20-sql-functions/17-table-functions/get-lineage.md @@ -0,0 +1,105 @@ +--- +title: GET_LINEAGE +--- + +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +返回表、视图、Stage 或列的上游或下游血缘。结果中的每一行表示血缘路径中的一条源对象到目标对象的关系。 + +在私有化部署中使用该函数之前，需要先在 `databend-query.toml` 中启用数据血缘。请参见[数据血缘](/guides/data-management/data-lineage#启用数据血缘)。 + +## 语法 + +```sql +GET_LINEAGE( + '', + '', + '' + [, ] +) +``` + +## 参数 + +| 参数 | 描述 | +|------|------| +| `object_name` | 查询起点。表或视图使用 `[catalog.]database.object`，Stage 使用 `stage_name`，列使用 `[catalog.]database.object.column`。省略 Catalog 或数据库时，将使用当前会话中的值。 | +| `object_domain` | 对象类型：`TABLE`、`VIEW`、`STAGE` 或 `COLUMN`。 | +| `direction` | `UPSTREAM` 表示向数据源方向追溯；`DOWNSTREAM` 表示向数据使用方方向追溯。 | +| `distance` | 可选的最大遍历跳数，取值范围为 `1` 到 `5`，默认为 `5`。 | + +所有参数均为位置参数。 + +## 输出列 + +| 列 | 类型 | 描述 | +|----|------|------| +| `source_object_catalog` | Nullable(String) | 源对象所属的 Catalog；Stage 为 `NULL`。 | +| `source_object_database` | Nullable(String) | 源对象所属的数据库；Stage 为 `NULL`。 | +| `source_object_name` | Nullable(String) | 源对象名称。 | +| `source_object_domain` | Nullable(String) | 源对象域：`TABLE`、`VIEW` 或 `STAGE`。 | +| `source_column_name` | Nullable(String) | 列级血缘中的源列；非列级血缘为 `NULL`。 | +| `source_status` | String | `ACTIVE`；如果源列应用了 Masking Policy，则为 `MASKED`。 | +| `target_object_catalog` | Nullable(String) | 目标对象所属的 Catalog；Stage 为 `NULL`。 | +| `target_object_database` | Nullable(String) | 目标对象所属的数据库；Stage 为 `NULL`。 | +| `target_object_name` | Nullable(String) | 目标对象名称。 | +| `target_object_domain` | Nullable(String) | 目标对象域：`TABLE`、`VIEW` 或 `STAGE`。 | +| `target_column_name` | Nullable(String) | 列级血缘中的目标列；非列级血缘为 `NULL`。 | +| `target_status` | String | `ACTIVE`；如果目标列应用了 Masking Policy，则为 `MASKED`。 | +| `distance` | Int32 | 相对于查询起点的跳数。直接关系的距离为 `1`。 | +| `process` | Nullable(String) | 创建该关系的操作元数据，采用 JSON 字符串格式，例如 Query ID、查询文本、用户、时间和血缘类型。 | + +## 示例 + +### 查询上游表 + +以下查询返回 `agg_customer_sales` 两跳以内的上游关系： + +```sql +SELECT + distance, + source_object_catalog, + source_object_database, + source_object_name, + source_object_domain, + target_object_database, + target_object_name +FROM GET_LINEAGE( + 'lineage_demo.agg_customer_sales', + 'TABLE', + 'UPSTREAM', + 2 +) +ORDER BY distance; +``` + +### 查询下游列 + +以下查询追踪 `fact_orders.amount` 被哪些列使用： + +```sql +SELECT + distance, + source_object_name, + source_column_name, + target_object_name, + target_column_name +FROM GET_LINEAGE( + 'lineage_demo.fact_orders.amount', + 'COLUMN', + 'DOWNSTREAM', + 5 +) +ORDER BY distance, target_object_name, target_column_name; +``` + +## 使用说明 + +- 如果对象存在但没有已记录的血缘，函数将返回空结果。 +- 查询结果会根据当前角色的对象可见性进行过滤。 +- Stage 仅支持对象级关系；Stage 文件字段不会作为稳定列返回。 +- `system` 和 `information_schema` 中的对象不会被记录为血缘源。 +- 外部 Catalog 对象会作为终止端点返回，不会继续向外部 Catalog 内部遍历。 +- 对于启用数据血缘之前已存在的视图，请使用 [`REFRESH LINEAGE`](/sql/sql-commands/ddl/view/refresh-lineage) 回填血缘关系。 diff --git a/docs/cn/sql-reference/20-sql-functions/17-table-functions/index.md b/docs/cn/sql-reference/20-sql-functions/17-table-functions/index.md index a706b23726..ff0638fef4 100644 --- a/docs/cn/sql-reference/20-sql-functions/17-table-functions/index.md +++ b/docs/cn/sql-reference/20-sql-functions/17-table-functions/index.md @@ -31,6 +31,7 @@ title: 表函数 (Table Functions) | [FUSE_VACUUM_TEMPORARY_TABLE](./fuse-vacuum-temporary-table.md) | 清理临时表 | `SELECT * FROM FUSE_VACUUM_TEMPORARY_TABLE()` | | [FUSE_AMEND](./fuse-amend.md) | 执行数据修正操作 | `SELECT * FROM FUSE_AMEND()` | | [TAG_REFERENCES](./tag-references.md) | 返回指定对象上分配的所有 Tag | `SELECT * FROM TAG_REFERENCES('default.users', 'TABLE')` | +| [GET_LINEAGE](./get-lineage.md) | 返回对象和列的上游或下游血缘 | `SELECT * FROM GET_LINEAGE('mydb.mytable', 'TABLE', 'UPSTREAM')` | ## Iceberg 集成函数 diff --git a/docs/en/guides/20-self-hosted/04-references/node-config/query-config.md b/docs/en/guides/20-self-hosted/04-references/node-config/query-config.md index 8e44d47b83..498d2eb47e 100644 --- a/docs/en/guides/20-self-hosted/04-references/node-config/query-config.md +++ b/docs/en/guides/20-self-hosted/04-references/node-config/query-config.md @@ -6,7 +6,7 @@ import FunctionDescription from '@site/src/components/FunctionDescription'; import LanguageDocs from '@site/src/components/LanguageDocs'; import DetailsWrap from '@site/src/components/DetailsWrap'; - + This page describes the Query node configurations available in the [databend-query.toml](https://github.com/databendlabs/databend/blob/main/scripts/distribution/configs/databend-query.toml) configuration file. @@ -77,6 +77,25 @@ When private task is enabled, `cloud_control_grpc_server_address` in the [query] | --------- | ---------------------------------------------------------------------------------------------------------------- | | on | Enables private task scheduling and execution on Databend Query nodes. Defaults to `false`. | +## [lineage] Section + +Use the `[lineage]` section to capture and persist object-level and column-level data lineage: + +```toml +[lineage] +on = true +# retention = 720 +``` + +Configure the same values on every Query node and restart the nodes after changing the configuration. When lineage is enabled, Databend configures its internal history storage automatically; do not add `lineage_history` to `[log.history.tables]`. + +| Parameter | Description | +|-----------|-------------| +| on | Enables lineage capture and persistence. Defaults to `false`. | +| retention | Optional retention period for DML lineage, in hours. If omitted, lineage is retained permanently. | + +For usage instructions, see [Data Lineage](/guides/data-management/data-lineage). + ## [log] Section This section can include these subsections: [log.file], [log.stderr], [log.query], and [log.tracing]. diff --git a/docs/en/guides/57-data-management/05-data-lineage.md b/docs/en/guides/57-data-management/05-data-lineage.md new file mode 100644 index 0000000000..d6b8f0f566 --- /dev/null +++ b/docs/en/guides/57-data-management/05-data-lineage.md @@ -0,0 +1,130 @@ +--- +title: Data Lineage +--- + +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +Data lineage shows how data moves from source objects to target objects. Use it to understand dependencies, assess the impact of a change, troubleshoot data pipelines, and trace a derived column back to its source. + +Databend records both object-level and column-level relationships: + +- **Upstream lineage** identifies the tables, views, or stages that supply data to an object. +- **Downstream lineage** identifies the objects that consume data from an object. +- **Column lineage** maps source columns to the derived target columns. + +![Table and column lineage in Databend Cloud](/img/guides/data-lineage.png) + +## Enable Data Lineage + +Databend Cloud manages lineage configuration for warehouses that provide the **Lineage** tab. For a self-hosted deployment, add the following section to `databend-query.toml` on every Query node, then restart the nodes: + +```toml title="databend-query.toml" +[lineage] +on = true +``` + +Lineage history is retained permanently by default. To retain it for a fixed number of hours, set `retention`, for example: + +```toml title="databend-query.toml" +[lineage] +on = true +retention = 720 +``` + +Use the dedicated `[lineage]` section only. Do not add the internal `lineage_history` table to `[log.history.tables]`. For all configuration options, see [Query Configurations](/guides/self-hosted/references/node-config/query-config#lineage-section). + +## Generate Lineage + +After lineage is enabled, Databend automatically records relationships created by operations such as `CREATE TABLE ... AS SELECT`, `CREATE VIEW`, `INSERT ... SELECT`, multi-table `INSERT`, `REPLACE`, `MERGE`, and `COPY`. Streams are resolved to their backing tables. + +The following example creates a two-hop lineage path: + +```sql +CREATE OR REPLACE DATABASE lineage_demo; + +CREATE OR REPLACE TABLE lineage_demo.fact_orders ( + order_id BIGINT, + customer_id BIGINT, + amount DECIMAL(12, 2), + order_time TIMESTAMP +); + +CREATE OR REPLACE TABLE lineage_demo.agg_customer_sales AS +SELECT + customer_id, + sum(amount) AS total_amount, + count(*) AS order_count, + max(order_time) AS last_order_time +FROM lineage_demo.fact_orders +GROUP BY customer_id; + +CREATE OR REPLACE TABLE lineage_demo.customer_segments AS +SELECT + customer_id, + total_amount, + order_count, + if(total_amount >= 1000, 'high_value', 'standard') AS segment, + now() AS updated_at +FROM lineage_demo.agg_customer_sales; +``` + +## Explore Lineage + +In Databend Cloud, open a table or view in Database Explorer and select the **Lineage** tab. The graph displays upstream and downstream objects, with column connections when column lineage is available. + +To retrieve lineage with SQL, use the [`GET_LINEAGE`](/sql/sql-functions/table-functions/get-lineage) table function: + +```sql +SELECT + distance, + source_object_database, + source_object_name, + target_object_database, + target_object_name +FROM GET_LINEAGE( + 'lineage_demo.agg_customer_sales', + 'TABLE', + 'UPSTREAM', + 2 +) +ORDER BY distance; +``` + +For column-level lineage, qualify the column name and use the `COLUMN` domain: + +```sql +SELECT + distance, + source_object_name, + source_column_name, + target_object_name, + target_column_name +FROM GET_LINEAGE( + 'lineage_demo.customer_segments.segment', + 'COLUMN', + 'UPSTREAM', + 2 +) +ORDER BY distance; +``` + +## Refresh Lineage for Existing Views + +Views created after lineage is enabled are tracked automatically. After enabling lineage on a deployment that already contains views, preview the missing or stale relationships and then refresh them: + +```sql +REFRESH LINEAGE FOR ALL VIEWS DRY RUN; +REFRESH LINEAGE FOR ALL VIEWS; +``` + +The refresh reconciles lineage for all views in the `default` catalog. It reports only views that need changes or could not be processed; unchanged views are omitted. The command requires the global `SUPER` privilege. See [`REFRESH LINEAGE`](/sql/sql-commands/ddl/view/refresh-lineage) for output details. + +## Limitations + +- `GET_LINEAGE` traverses at most five hops. +- System and `information_schema` objects are excluded as lineage sources. +- Stages participate in object-level lineage, but staged file fields do not provide stable column-level mappings. +- External-catalog objects can appear as endpoints but are not traversed beyond the external catalog boundary. +- Results include only objects visible to the current role. diff --git a/docs/en/guides/57-data-management/index.md b/docs/en/guides/57-data-management/index.md index 7b0d5e4167..9c3f4d4c43 100644 --- a/docs/en/guides/57-data-management/index.md +++ b/docs/en/guides/57-data-management/index.md @@ -10,3 +10,4 @@ title: Data Management | **[Data Recovery](./02-data-recovery.md)** | Access and restore past data | • Time Travel
• Flashback Tables
• Backup & Restore
• AT & UNDROP | • SELECT ... AT
• FLASHBACK TABLE
• BENDSAVE BACKUP | | **[Data Protection](./03-data-protection.md)** | Secure access and prevent loss | • Network Policies
• Access Control
• Time Travel & Fail-safe
• Data Encryption | • NETWORK POLICY
• GRANT/REVOKE
• USER/ROLE | | **[Data Recycle](./04-data-recycle.md)** | Free up storage space | • VACUUM Commands
• Retention Policies
• Orphan File Cleanup
• Temporary File Management | • VACUUM TABLE
• VACUUM DROP TABLE
• DATA_RETENTION_TIME | +| **[Data Lineage](./05-data-lineage.md)** | Trace data flow and dependencies | • Upstream & Downstream
• Object & Column Lineage
• Impact Analysis | • GET_LINEAGE
• REFRESH LINEAGE
• Lineage Graph | diff --git a/docs/en/sql-reference/10-sql-commands/00-ddl/05-view/index.md b/docs/en/sql-reference/10-sql-commands/00-ddl/05-view/index.md index 0ead37a086..7537babfbc 100644 --- a/docs/en/sql-reference/10-sql-commands/00-ddl/05-view/index.md +++ b/docs/en/sql-reference/10-sql-commands/00-ddl/05-view/index.md @@ -12,6 +12,7 @@ This page provides a comprehensive overview of view operations in Databend, orga | [ALTER VIEW](ddl-alter-view.md) | Assigns or removes tags on an existing view | | [DROP VIEW](ddl-drop-view.md) | Removes a view | | [Materialized Views](materialized-view.md) | Creates and maintains a materialized view backed by physical storage | +| [REFRESH LINEAGE](refresh-lineage.md) | Backfills or reconciles lineage for existing views | ## View Information diff --git a/docs/en/sql-reference/10-sql-commands/00-ddl/05-view/refresh-lineage.md b/docs/en/sql-reference/10-sql-commands/00-ddl/05-view/refresh-lineage.md new file mode 100644 index 0000000000..4c7e9e3b48 --- /dev/null +++ b/docs/en/sql-reference/10-sql-commands/00-ddl/05-view/refresh-lineage.md @@ -0,0 +1,71 @@ +--- +title: REFRESH LINEAGE +sidebar_position: 7 +--- + +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +Backfills or reconciles lineage for existing views in the `default` catalog. Use this command after enabling data lineage on a deployment that already contains views. Views created after lineage is enabled are tracked automatically. + +This command requires the global `SUPER` privilege and lineage must be enabled. See [Data Lineage](/guides/data-management/data-lineage#enable-data-lineage). + +## Syntax + +```sql +REFRESH LINEAGE FOR ALL VIEWS [ DRY RUN ] +``` + +`DRY RUN` calculates and reports the changes without writing them. Run it first to review the work that a refresh would perform. + +## Output Columns + +| Column | Description | +|--------|-------------| +| `object_domain` | Object domain; currently `VIEW`. | +| `catalog` | Catalog containing the view; currently `default`. | +| `database` | Database containing the view. | +| `object_name` | View name. | +| `status` | `DRY_RUN`, `REFRESHED`, or `ERROR`. | +| `edge_count` | Number of lineage edges found in the current view definition. | +| `upsert_count` | Number of missing or changed edges to add or update. | +| `delete_count` | Number of stale edges to remove. | +| `error` | Error details when `status` is `ERROR`; otherwise `NULL`. | + +Successful views with no changes are omitted from the result. + +## Examples + +Preview the changes required for existing views: + +```sql +REFRESH LINEAGE FOR ALL VIEWS DRY RUN; +``` + +Apply the changes: + +```sql +REFRESH LINEAGE FOR ALL VIEWS; +``` + +After the command completes, query a view's upstream lineage with [`GET_LINEAGE`](/sql/sql-functions/table-functions/get-lineage): + +```sql +SELECT + distance, + source_object_database, + source_object_name, + target_object_database, + target_object_name +FROM GET_LINEAGE( + 'lineage_demo.sales_view', + 'VIEW', + 'UPSTREAM', + 1 +); +``` + +:::note +To change a logical view definition, use [`CREATE OR REPLACE VIEW`](ddl-create-view.md). `ALTER VIEW ... AS ...` is not supported because changing the definition without recreating the view could leave persisted lineage inconsistent. +::: diff --git a/docs/en/sql-reference/20-sql-functions/17-table-functions/get-lineage.md b/docs/en/sql-reference/20-sql-functions/17-table-functions/get-lineage.md new file mode 100644 index 0000000000..bece09f1df --- /dev/null +++ b/docs/en/sql-reference/20-sql-functions/17-table-functions/get-lineage.md @@ -0,0 +1,105 @@ +--- +title: GET_LINEAGE +--- + +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +Returns upstream or downstream lineage for a table, view, stage, or column. Each returned row represents one source-to-target relationship in the lineage path. + +Before using this function in a self-hosted deployment, enable lineage in `databend-query.toml`. See [Data Lineage](/guides/data-management/data-lineage#enable-data-lineage). + +## Syntax + +```sql +GET_LINEAGE( + '', + '', + '' + [, ] +) +``` + +## Arguments + +| Argument | Description | +|----------|-------------| +| `object_name` | Object to start from. Use `[catalog.]database.object` for a table or view, `stage_name` for a stage, and `[catalog.]database.object.column` for a column. Names that omit the catalog or database use the current session values. | +| `object_domain` | Object type: `TABLE`, `VIEW`, `STAGE`, or `COLUMN`. | +| `direction` | `UPSTREAM` traces toward sources; `DOWNSTREAM` traces toward consumers. | +| `distance` | Optional maximum number of hops to traverse, from `1` to `5`. Defaults to `5`. | + +Arguments are positional. + +## Output Columns + +| Column | Type | Description | +|--------|------|-------------| +| `source_object_catalog` | Nullable(String) | Catalog containing the source object; `NULL` for a stage. | +| `source_object_database` | Nullable(String) | Database containing the source object; `NULL` for a stage. | +| `source_object_name` | Nullable(String) | Name of the source object. | +| `source_object_domain` | Nullable(String) | Domain of the source object: `TABLE`, `VIEW`, or `STAGE`. | +| `source_column_name` | Nullable(String) | Source column for column lineage; otherwise `NULL`. | +| `source_status` | String | `ACTIVE`, or `MASKED` when the source column has a masking policy. | +| `target_object_catalog` | Nullable(String) | Catalog containing the target object; `NULL` for a stage. | +| `target_object_database` | Nullable(String) | Database containing the target object; `NULL` for a stage. | +| `target_object_name` | Nullable(String) | Name of the target object. | +| `target_object_domain` | Nullable(String) | Domain of the target object: `TABLE`, `VIEW`, or `STAGE`. | +| `target_column_name` | Nullable(String) | Target column for column lineage; otherwise `NULL`. | +| `target_status` | String | `ACTIVE`, or `MASKED` when the target column has a masking policy. | +| `distance` | Int32 | Number of hops from the requested object. A direct relationship has distance `1`. | +| `process` | Nullable(String) | JSON-formatted metadata about the operation that created the relationship, such as its query ID, query text, user, time, and lineage kind. | + +## Examples + +### Find Upstream Tables + +This query returns up to two upstream hops for `agg_customer_sales`: + +```sql +SELECT + distance, + source_object_catalog, + source_object_database, + source_object_name, + source_object_domain, + target_object_database, + target_object_name +FROM GET_LINEAGE( + 'lineage_demo.agg_customer_sales', + 'TABLE', + 'UPSTREAM', + 2 +) +ORDER BY distance; +``` + +### Find Downstream Columns + +This query traces where `fact_orders.amount` is used: + +```sql +SELECT + distance, + source_object_name, + source_column_name, + target_object_name, + target_column_name +FROM GET_LINEAGE( + 'lineage_demo.fact_orders.amount', + 'COLUMN', + 'DOWNSTREAM', + 5 +) +ORDER BY distance, target_object_name, target_column_name; +``` + +## Usage Notes + +- If the object exists but has no recorded lineage, the function returns no rows. +- Results are filtered according to the current role's object visibility. +- Stage relationships are object-level only; staged file fields are not returned as stable columns. +- System and `information_schema` objects are not recorded as lineage sources. +- External-catalog objects are returned as terminal endpoints and are not traversed further. +- Use [`REFRESH LINEAGE`](/sql/sql-commands/ddl/view/refresh-lineage) to backfill lineage for views that existed before lineage was enabled. diff --git a/docs/en/sql-reference/20-sql-functions/17-table-functions/index.md b/docs/en/sql-reference/20-sql-functions/17-table-functions/index.md index 615c76b400..699c134fc4 100644 --- a/docs/en/sql-reference/20-sql-functions/17-table-functions/index.md +++ b/docs/en/sql-reference/20-sql-functions/17-table-functions/index.md @@ -40,6 +40,7 @@ This page provides reference information for the table functions in Databend. Ta | [TASK_HISTROY](./task_histroy.md) | Shows task execution history | `SELECT * FROM TASK_HISTROY('mytask')` | | [POLICY_REFERENCES](./policy-references.md) | Returns associations between security policies and tables/views | `SELECT * FROM POLICY_REFERENCES(POLICY_NAME => 'mypolicy')` | | [TAG_REFERENCES](./tag-references.md) | Returns tags assigned to a database object | `SELECT * FROM TAG_REFERENCES('mydb.mytable', 'TABLE')` | +| [GET_LINEAGE](./get-lineage.md) | Returns upstream or downstream object and column lineage | `SELECT * FROM GET_LINEAGE('mydb.mytable', 'TABLE', 'UPSTREAM')` | ## Storage Engine Functions diff --git a/static/img/guides/data-lineage.png b/static/img/guides/data-lineage.png new file mode 100644 index 0000000000..8a827e0746 Binary files /dev/null and b/static/img/guides/data-lineage.png differ