morrySnow commented on code in PR #3997: URL: https://github.com/apache/doris-website/pull/3997#discussion_r3619018342
########## community/developer-guide/data-lineage-plugin-development.md: ########## @@ -0,0 +1,604 @@ +--- +title: Data Lineage Plugin Development +language: en +description: Build, package, deploy, and verify an Apache Doris data lineage plugin that logs lineage events or forwards them to an external governance system. +keywords: + - Apache Doris + - data lineage + - LineagePlugin + - LineagePluginFactory + - ServiceLoader + - plugin development +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +<!-- Knowledge type: Extension development --> +<!-- Applicable scenario: Data governance integration / Kernel extension development --> + +# Data Lineage Plugin Development + +This document describes how to develop an external data lineage plugin for Apache Doris. The plugin receives a `LineageInfo` event after a supported DML statement succeeds, then converts the event and writes it to logs, a messaging system, or an external metadata governance platform. + +> Applicable version: Apache Doris community edition 4.0.6 and later. Compile the plugin against the exact Doris community release or source revision deployed on the FE. + +## Scope and lifecycle + +The lineage framework is an FE-side SPI. It does not store lineage data, provide a query API, or render a lineage graph. A plugin converts the in-memory Java objects in `LineageInfo` to the format and transport protocol required by its downstream system. + +The current implementation generates events only for successful `INSERT INTO ... SELECT`, `INSERT OVERWRITE TABLE ... SELECT`, and `CREATE TABLE AS SELECT` statements. `VALUES`-only writes and writes whose target is `__internal_schema` are skipped. `SELECT`, `UPDATE`, `DELETE`, and load jobs are outside the scope of this framework. Some `UPDATE` and `DELETE` execution paths internally reuse an Insert command, but the original command-type check excludes them from lineage event submission. + + + +`eventFilter()` is called once before lineage extraction on the DML query path and again before worker dispatch. Return `false` when the plugin should not receive events. The method is called concurrently from query threads and the worker thread, so it must be thread-safe. `exec()` is called by one worker thread, but can run concurrently with `eventFilter()`. + +At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` exposes an immutable property map through `getProperties()`; FE stores the plugin name and directory in the `plugin.name` and `plugin.path` entries. Access them with `context.getProperties().get("plugin.name")` and `context.getProperties().get("plugin.path")`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through the `plugin.path` entry, and initialize its resources there. + +## Lineage event model + +`LineageInfo` contains a target table, target output columns, source tables, direct lineage, indirect lineage, and a `LineageContext`. + +| Part | API | Meaning | +| --- | --- | --- | +| Target | `getTargetTable()`, `getTargetColumns()` | The target table and output columns of the write. | +| Source tables | `getTableLineageSet()` | Tables referenced by the analyzed logical plan. | +| Direct lineage | `getDirectLineageMap()` | A map from each output Slot to source expressions, classified as `IDENTITY`, `TRANSFORMATION`, or `AGGREGATION`. | +| Dataset indirect lineage | `getDatasetIndirectLineageMap()` | Expressions that affect the complete result set: `JOIN`, `FILTER`, `GROUP_BY`, and `SORT`. | +| Output indirect lineage | `getOutputIndirectLineageMap()` | Dependencies that affect a specific output column: `WINDOW` and `CONDITIONAL`. | +| Query context | `getContext()` | Query ID, SQL text, user, client IP, session database and catalog, execution state, timestamp, duration, and sanitized external catalog properties. | Review Comment: 要先有一个summary，类似你这个。然后要有一个detail，每个接口逐一介绍，写清楚接口的输入输出类型。可以看看 javadoc 接口是怎么写的，比如：https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#containsKey-java.lang.Object- ########## community/developer-guide/data-lineage-plugin-development.md: ########## @@ -0,0 +1,604 @@ +--- +title: Data Lineage Plugin Development +language: en +description: Build, package, deploy, and verify an Apache Doris data lineage plugin that logs lineage events or forwards them to an external governance system. +keywords: + - Apache Doris + - data lineage + - LineagePlugin + - LineagePluginFactory + - ServiceLoader + - plugin development +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +<!-- Knowledge type: Extension development --> +<!-- Applicable scenario: Data governance integration / Kernel extension development --> + +# Data Lineage Plugin Development + +This document describes how to develop an external data lineage plugin for Apache Doris. The plugin receives a `LineageInfo` event after a supported DML statement succeeds, then converts the event and writes it to logs, a messaging system, or an external metadata governance platform. + +> Applicable version: Apache Doris community edition 4.0.6 and later. Compile the plugin against the exact Doris community release or source revision deployed on the FE. + +## Scope and lifecycle + +The lineage framework is an FE-side SPI. It does not store lineage data, provide a query API, or render a lineage graph. A plugin converts the in-memory Java objects in `LineageInfo` to the format and transport protocol required by its downstream system. + +The current implementation generates events only for successful `INSERT INTO ... SELECT`, `INSERT OVERWRITE TABLE ... SELECT`, and `CREATE TABLE AS SELECT` statements. `VALUES`-only writes and writes whose target is `__internal_schema` are skipped. `SELECT`, `UPDATE`, `DELETE`, and load jobs are outside the scope of this framework. Some `UPDATE` and `DELETE` execution paths internally reuse an Insert command, but the original command-type check excludes them from lineage event submission. + + + +`eventFilter()` is called once before lineage extraction on the DML query path and again before worker dispatch. Return `false` when the plugin should not receive events. The method is called concurrently from query threads and the worker thread, so it must be thread-safe. `exec()` is called by one worker thread, but can run concurrently with `eventFilter()`. + +At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` exposes an immutable property map through `getProperties()`; FE stores the plugin name and directory in the `plugin.name` and `plugin.path` entries. Access them with `context.getProperties().get("plugin.name")` and `context.getProperties().get("plugin.path")`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through the `plugin.path` entry, and initialize its resources there. + +## Lineage event model + +`LineageInfo` contains a target table, target output columns, source tables, direct lineage, indirect lineage, and a `LineageContext`. + +| Part | API | Meaning | +| --- | --- | --- | +| Target | `getTargetTable()`, `getTargetColumns()` | The target table and output columns of the write. | +| Source tables | `getTableLineageSet()` | Tables referenced by the analyzed logical plan. | +| Direct lineage | `getDirectLineageMap()` | A map from each output Slot to source expressions, classified as `IDENTITY`, `TRANSFORMATION`, or `AGGREGATION`. | +| Dataset indirect lineage | `getDatasetIndirectLineageMap()` | Expressions that affect the complete result set: `JOIN`, `FILTER`, `GROUP_BY`, and `SORT`. | +| Output indirect lineage | `getOutputIndirectLineageMap()` | Dependencies that affect a specific output column: `WINDOW` and `CONDITIONAL`. | +| Query context | `getContext()` | Query ID, SQL text, user, client IP, session database and catalog, execution state, timestamp, duration, and sanitized external catalog properties. | + +Direct lineage describes how an output value is produced. A plain source-column reference is `IDENTITY`, for example `target.customer_id <- source.customer_id`. A calculation, function, window expression, or conditional expression without an aggregate function is `TRANSFORMATION`, for example `UPPER(source.region)`. An expression that contains an aggregate function is `AGGREGATION`, for example `SUM(source.amount)`. Indirect lineage records expressions that affect the output without directly becoming an output value, including Join keys, `WHERE` and `HAVING` predicates, grouping keys, and sorting keys. `WINDOW` records window partitioning and ordering inputs. `CONDITIONAL` records the effect of `CASE`, `IF`, or `COALESCE` on a specific output column. Review Comment: 表格描述 ########## community/developer-guide/data-lineage-plugin-development.md: ########## @@ -0,0 +1,604 @@ +--- +title: Data Lineage Plugin Development +language: en +description: Build, package, deploy, and verify an Apache Doris data lineage plugin that logs lineage events or forwards them to an external governance system. +keywords: + - Apache Doris + - data lineage + - LineagePlugin + - LineagePluginFactory + - ServiceLoader + - plugin development +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +<!-- Knowledge type: Extension development --> +<!-- Applicable scenario: Data governance integration / Kernel extension development --> + +# Data Lineage Plugin Development + +This document describes how to develop an external data lineage plugin for Apache Doris. The plugin receives a `LineageInfo` event after a supported DML statement succeeds, then converts the event and writes it to logs, a messaging system, or an external metadata governance platform. + +> Applicable version: Apache Doris community edition 4.0.6 and later. Compile the plugin against the exact Doris community release or source revision deployed on the FE. + +## Scope and lifecycle + +The lineage framework is an FE-side SPI. It does not store lineage data, provide a query API, or render a lineage graph. A plugin converts the in-memory Java objects in `LineageInfo` to the format and transport protocol required by its downstream system. + +The current implementation generates events only for successful `INSERT INTO ... SELECT`, `INSERT OVERWRITE TABLE ... SELECT`, and `CREATE TABLE AS SELECT` statements. `VALUES`-only writes and writes whose target is `__internal_schema` are skipped. `SELECT`, `UPDATE`, `DELETE`, and load jobs are outside the scope of this framework. Some `UPDATE` and `DELETE` execution paths internally reuse an Insert command, but the original command-type check excludes them from lineage event submission. + + + +`eventFilter()` is called once before lineage extraction on the DML query path and again before worker dispatch. Return `false` when the plugin should not receive events. The method is called concurrently from query threads and the worker thread, so it must be thread-safe. `exec()` is called by one worker thread, but can run concurrently with `eventFilter()`. + +At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` exposes an immutable property map through `getProperties()`; FE stores the plugin name and directory in the `plugin.name` and `plugin.path` entries. Access them with `context.getProperties().get("plugin.name")` and `context.getProperties().get("plugin.path")`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through the `plugin.path` entry, and initialize its resources there. + +## Lineage event model Review Comment: 这个章节叫做 event model，下面的接口介绍，看起来和他无关。应当单独放置？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
