davidradl commented on code in PR #25762: URL: https://github.com/apache/flink/pull/25762#discussion_r1875799164
########## docs/content/docs/internals/data_lineage.md: ########## @@ -0,0 +1,74 @@ +--- +title: Data Lineage +weight: 12 +type: docs +aliases: + - /internals/data_lineage.html +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Native Lineage Support +Data lineage has gain more and more criticality in data ecosystem. As Apache Flink is widely used for data ingestion and ETL in Streaming Data Lake, we need +an end to end lineage solution for scenarios including but not limited to: + - `Data Quality Assurance`: Identifying and rectifying data inconsistencies by tracing data errors back to their origin within the data pipeline. + - `Data Governance`： Establishing clear data ownership and accountability by documenting data origins and transformations. + - `Regulatory Compliance`: Ensuring adherence to data privacy and compliance regulations by tracking data flow and transformations throughout its lifecycle. + - `Data Optimization`: Identifying redundant data processing steps and optimizing data flows to improve efficiency. + +Apache Flink provides a native lineage support for the community requirement by providing an internal lineage data model and [Job Status Listener]({{< ref "docs/deployment/advanced/job_status_listener" >}}) for +developer to integrate lineage metadata into external lineage system, for example [OpenLineage](https://openlineage.io). When a job is created in Flink runtime, the JobCreatedEvent +contains the Lineage Graph metadata will be sent to Job Status Listeners. + +# Lineage Data Model +Flink native lineage interfaces are defined in two layers. The first layer is the generic interface for all Flink jobs and connector, and the second layer defines +the extended interfaces for Table and DataStream independently. The interface and class relationship are defined in the diagram below. + +{{< img src="/fig/lineage_interfaces.png" alt="Lineage Data Model" width="80%">}} + +By default, Table related lineage interfaces or classes are mainly used in Flink Table Runtime, thus Flink users doesn't need to touch these interfaces. Flink community will gradually support all +of common connectors, such as Kafka, JDBC, Cassandra, Hive and so on. If you have customized connector defined, you need to have customized source/sink implements the LineageVertexProvider interface. +Within a LineageVertex, a list of Lineage Dataset is defined as metadata for Flink source/sink. Review Comment: NITs: Lineage Dataset is -> Lineage Datasets are for Flink -> for the Flink -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
