We are planning to do this session next Thursday at 5pm CET 9am PT. I will send a zoom link in advance. Julien
On Sat, Feb 25, 2023 at 05:59 Jarek Potiuk <ja...@potiuk.com> wrote: > Cool. I am looking forward to it :). It would be great to get some > insight from those who attempted to get the lineage working in several > versions of Open Lineage and finally arrived at the current > specs/integration. > > On Wed, Feb 22, 2023 at 7:02 PM Julien Le Dem > <jul...@astronomer.io.invalid> wrote: > > > > Thank you Jarek, > > I am happy to organize a zoom presentation about OpenLineage and answer > any question. It is indeed a spec decoupling the data transformation layer > from the Metadata store people are using. Just like OpenTelemetry is for > service metrics/traces. > > Best, > > Julien > > > > On Tue, Feb 21, 2023 at 11:23 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >> And to add a little "parallel" - I think Open Lineage integration > replacing our "generic lineage" is very similar step to the new > "Multi-tenant"-ready authentication interface we are discussing in > https://lists.apache.org/thread/cc9dj680nwz494k8n51w6qqohzm4wgck > >> > >> Yes - we have a generic authentication interface, but no - it's useless > for the case where multi-tenancy and good level of resource authorization > is needed. It's just far too simplistic and limited. > >> > >> Same with current lineage generic interface - yes, we have it but it's > only useful in a limited set of cases. and if we want to step-it-up we need > to come up with something better (and Open Lineage happens to be one that > has been developed with Airflow in mind and battle tested). > >> > >> J. > >> > >> On Wed, Feb 22, 2023 at 8:16 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >>> > >>> Hey Rafał (Eugene, Michal - and others who are looking), > >>> > >>> I think I know where your/Eugen/Michał concerns are coming from. And I > think it would be great if we can talk it over a bit. I believe this is - > in parts - quite a misunderstanding of what Open Lineage really is, how > much of an integration it is and what are the reasons why it has been > implemented the way it was implemented in Airflow. > >>> > >>> **Idea**: (Julien - Maybe you can organize it ?): > >>> > >>> Maybe we can have an open-to-everyone presentation/zoom call with > quite some time foreseen to ask questions where you would explain the > community about those integration points (and especially those people who > are worried we are losing something by choosing the OpenLineage > integration). I would love to see such a presentation - specifically > focused on explaining how Open-Lineage is really improving the current > lineage approach and what problems it solves that the existing generic > interface doesn't. > >>> > >>> Just to set the tone and focus for such meeting if we have one: > >>> > >>> For me - when I look at Open Lineage, it is really "this is how > lineage generic interface **should** be done in Airflow". The "generic" > lineage support we have now is very, very basic, I'd even say far too > simplistic. I would even say, it's useless besides a few, very basic use > cases. Simply because there was never a good "receiver" of the information > to cover those cases. > >>> > >>> When you look closely at OpenLineage, it's nothing more than a better > convention of the dictionaries that we send as a metadata, better meta-data > in case of SQL operators (Hooks in the future hopefully), allowing handling > some cases that current lineage simply cannot. Also what open-lineage > integration with Airflow covers better handling of the lifecycle "task" and > "dag" in Airflow to be able to bind lineage data together. That's my > understanding of what we get when we integrate OL in. > >>> > >>> I think over the last 2 years Datakin/Astronomer people had worked out > the level of interface that **just works** and if we would like to get the > lineage information from Airflow as useful as it is in OL, we would have to > anyway implement pretty much all of the things they already did. > >>> > >>> I would love (and I think many community members) to take part in such > a call to hear on that particular aspect of the OL integration. > >>> > >>> J. > >>> > >>> On Wed, Feb 22, 2023 at 12:40 AM Rafal Biegacz < > rafalbieg...@google.com.invalid> wrote: > >>>> > >>>> Hi, > >>>> > >>>> I second/echo the input provided by Eugene and Michal. > >>>> > >>>> In general, Airflow should provide generic interfaces to lineage > backends so it's easy to configure the one preferred by the user. Whether > it's Open Lineage, proprietary solution, Dataplex Lineage, etc. it should > be the user's choice. > >>>> > >>>> We should avoid close integration with any specific lineage backend > due to the reasons already mentioned, i.e. to avoid translations between > lineage backends. Also, we would closely couple one framework (Airflow) > with another one (Open Lineage) - it makes Airflow more complex and less > flexible. Loose coupling between lineage backends and Airflow seems to be > more future-proven. > >>>> > >>>> Regards, Rafal. > >>>> > >>>> > >>>> On Sat, Feb 11, 2023 at 12:21 AM Julien Le Dem > <jul...@astronomer.io.invalid> wrote: > >>>>> > >>>>> Dear Airflow community, > >>>>> I have transferred the content of the working google doc I shared a > few weeks ago to the Airflow confluence: > >>>>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-53+OpenLineage+in+Airflow > >>>>> All comments have been answered, I added clarifications to the doc > accordingly and I also added your suggestions to improve the proposal. > >>>>> All that history is linked from the discussion thread link in the > confluence doc if you wish to consult it. > >>>>> Thank you all for your feedback and help in the process. > >>>>> Best > >>>>> Julien > >>>>> > >>>>> > >>>>> On Fri, Feb 10, 2023 at 2:55 PM Julien Le Dem <jul...@astronomer.io> > wrote: > >>>>>> > >>>>>> Thank you for the email Jarek, and Eugene for your suggestions, > >>>>>> I do agree with Jarek's assessment. I don't have very much to add > to his argument, it is very thoughtful! > >>>>>> OpenLineage was started to avoid the cartesian complexity that > Eugene mentions. There's actually that specific illustration in the > OpenLineage doc. > >>>>>> Lineage consumers want to avoid having to understand the lineage > format of each individual observed data transformation layer. And > transformation layers don't want to understand every Metadata store's model > and protocol. > >>>>>> Eugene, about your specific proposal about a global vocabulary of > entities, I think it is a great suggestion. > >>>>>> We can map those entities to Datasets in OpenLineage. The way > OpenLineage models this is by allowing specific facets attached to Dataset. > Facets are pieces of metadata each with their own JsonSchema. > >>>>>> For example a table from a relational database will have a schema > facet when a file in GCS might not. > >>>>>> So I think in Airflow we could have each of the entity classes you > describe be used in the get_openlineage_facets*() API in the Operators. > >>>>>> Each of those classes would know what OpenLineage facets they can > expose. > >>>>>> I'll add a mention in the AIP and I think we can go in more details > in a ticket. > >>>>>> Cheers, > >>>>>> Julien > >>>>>> > >>>>>> On Fri, Feb 10, 2023 at 12:27 PM Jarek Potiuk <ja...@potiuk.com> > wrote: > >>>>>>> > >>>>>>> Just a quick personal view on it, Eugene (I bet Julian's answer > will > >>>>>>> be more thoughtful). > >>>>>>> > >>>>>>> I think you are right to the "agnostic" part. But I have one > question > >>>>>>> - what are we considering "agnostic"? > >>>>>>> > >>>>>>> There is no "widespread" standard for lineage (yet). Open Lineage > >>>>>>> with its donation to Linux Foundation Data & AI is aspiring to > become > >>>>>>> one. And it's a pretty good candidate: > >>>>>>> > >>>>>>> * designed from grounds-up to be agnostic (Open Lineage was only > >>>>>>> published as an API from day one) > >>>>>>> * as of recently, the ownership and governance of Open Lineage is > with > >>>>>>> Linux Foundation Data & AI (https://lfaidata.foundation/) which > is > >>>>>>> part of "Linux Foundation Project" - well known and respectful > >>>>>>> foundation that - similarly to the ASF is an umbrella and provides > >>>>>>> governance rules for a big number of well established OSS projects > >>>>>>> > >>>>>>> In essence it is the same approach as we already discussed and > >>>>>>> approved for Open Telemetry (which is governed by CNCF which is in > the > >>>>>>> same league as recognition and governance to LFP) (not yet > implemented > >>>>>>> though). In the case of Open-Telemetry, we decided against > developing > >>>>>>> our "own" existing standard but we opted for one that is out there. > >>>>>>> Yes it is a bit more established and popular than Open Lineage is, > but > >>>>>>> i so wish that we chose and implemented it already (and earlier as > not > >>>>>>> having a standard there - except statsd which is really, really > poor) > >>>>>>> has a great impact on Airflow being just "pluggable" in existing > >>>>>>> solutions for monitoring. (BTW. I hope we implement it soon and I > hear > >>>>>>> (and see) there are attempts to do so). > >>>>>>> > >>>>>>> In the case of Open Lineage, the questions are - is there an > >>>>>>> alternative of the same caliber? Shall we produce our own "agnostic > >>>>>>> standard" for it instead ? Is there a chance the idea of > >>>>>>> "airflow-specific" attributes will catch up and many "consumers" > will > >>>>>>> be writing their own conversions to the way they can consume it? > >>>>>>> > >>>>>>> I would really, really try to avoid the pitfalls nicely summarized > >>>>>>> here: https://xkcd.com/927/ > >>>>>>> > >>>>>>> We can of course make a wrong bet and in 2 years Airflow might be > the > >>>>>>> only one supporting Open Lineage. That might happen. Though the > list > >>>>>>> of "consumers" of Open Lineage is already pretty good IMHO. Or > maybe - > >>>>>>> more likely - once Airflow implements it, due to Airflow's > popularity > >>>>>>> and the fact that there is already competition supporting it (e.g. > >>>>>>> Amundsen) we will increase the chance of "hockey-stick" adoption of > >>>>>>> Open Lineage. My bet is - the latter and for the benefit of the > whole > >>>>>>> ecosystem. I think we have a chance to influence creation of a new, > >>>>>>> important standard. Much less so, I think if we just provide our > own > >>>>>>> custom solution - with lots and lots of work for others to be able > to > >>>>>>> consume it, no time to properly nurture the API and make it easier > to > >>>>>>> implement it (which is undoubtedly what Datakin, Astronomer and now > >>>>>>> LFData & AI run governance main focus is) > >>>>>>> > >>>>>>> Are there other alternatives we should consider ? Do we want to > >>>>>>> develop our own standard (and implement all the integrations from > the > >>>>>>> grounds up) ? > >>>>>>> > >>>>>>> J. > >>>>>>> > >>>>>>> On Fri, Feb 10, 2023 at 11:40 AM Eugen Kosteev <eu...@kosteev.com> > wrote: > >>>>>>> > > >>>>>>> > Hi Julien. > >>>>>>> > > >>>>>>> > I reviewed the design doc. > >>>>>>> > The general idea looks good to me, but I have some concerns that > I would like to share. > >>>>>>> > > >>>>>>> > If I understand correctly the proposed design is to fill in > "operators" with self-methods to extract lineage metadata from it, and I > agree with the motivation. If those are decoupled (in a form of extractors > in separate package) from operators itself, then the downsides is that (as > you mentioned) - extractors will be distributed separately and "operators" > logic is out of sync with "lineage extraction" logic by design. > >>>>>>> > Also knowledge about internals of operator spills out of the > operator which is not good at all (at the very least). > >>>>>>> > > >>>>>>> > However, if we make every operator being exposing method to > generate lineage metadata of the specific format, e.g. OpenLineage etc., > then we will end up with cartesian complexity of supporting in each > provider+operator each backend format. > >>>>>>> > > >>>>>>> > If you say that the goal is that "operators" will always > generate OpenLineage format only and each consumer will convert this format > to their own internal representation, well, if they do this then this seems > like a working approach. But with the assumption that each consumer will > support it. > >>>>>>> > > >>>>>>> > I think it comes down to the question: is OpenLineage format > enough popular, complete and proper for the lineage metadata that every > consumer will be convinced to support it. We may also consider issues like > mismatch of lineage feature parity, e.g. OpenLineage supports field-level > lineage but consumer doesn't support (or not at the moment), so we would > prefer lineage metadata transferred to the backend to be slightly different > in this case. > >>>>>>> > > >>>>>>> > What do you think about the idea: > >>>>>>> > 1. make lineage metadata generated by "operators" to be agnostic > of the specific format, just using entities from big generic vocabulary of > entities e.g. created here > https://github.com/apache/airflow/blob/main/airflow/lineage/entities.py. > We would have there e.g. entities like: > >>>>>>> > > -------------------------------------------------------------------- > >>>>>>> > @attr.s(auto_attribs=True, kw_only=True) > >>>>>>> > class PostgresTable: > >>>>>>> > """Airflow lineage entity representing Postgres table.""" > >>>>>>> > > >>>>>>> > host: str = attr.ib() > >>>>>>> > port: str = attr.ib() > >>>>>>> > database: str = attr.ib() > >>>>>>> > schema: str = attr.ib() > >>>>>>> > table: str = attr.ib() > >>>>>>> > > >>>>>>> > @attr.s(auto_attribs=True, kw_only=True) > >>>>>>> > class GCSEntity: > >>>>>>> > """Airflow lineage entity representing generic Google Cloud > Storage entity.""" > >>>>>>> > > >>>>>>> > bucket: str = attr.ib() > >>>>>>> > path: str = attr.ib() > >>>>>>> > > >>>>>>> > @attr.s(auto_attribs=True, kw_only=True) > >>>>>>> > class AWSS3Entity: > >>>>>>> > """Airflow lineage entity representing generic AWS S3 > entity.""" > >>>>>>> > > >>>>>>> > bucket: str = attr.ib() > >>>>>>> > path: str = attr.ib() > >>>>>>> > > -------------------------------------------------------------------- > >>>>>>> > 2. Implement "adapters" that will act as a bridge between > "operators" and backends. Their responsibility will be to convert lineage > metadata generated by "operators" to a format understandable by specific > backend. > >>>>>>> > And then we can use the built-in mechanism of inlets/outlets to > bypass Airflow lineage metadata to the Airflow lineage backend. > >>>>>>> > > >>>>>>> > I didn't get exactly implementation details of your proposed > design, but I think maintaining global vocabulary of entities to use in > inlets/outlets of operators is crucial for Airflow, as this could be > leveraged to build various features on top of it, like displaying lineage > graph in Airflow UI (based on XCOM):) > >>>>>>> > > >>>>>>> > Importantly to note, if we decide to send out from Airflow > lineage metadata only in OpenLineage format, well, we could have than only > one "adapter" OpenLineageAdapter. But the "adapters" approach leaves us > room for adding support to others (following "pluggable" approach as > Airflow is mainly known/good about). > >>>>>>> > > >>>>>>> > All in all: > >>>>>>> > - global vocabulary of entities used across all "operators" > (with all advantages out of it, mentioned above) > >>>>>>> > - "adapters" approach > >>>>>>> > seems to me crucial points in the design that make sense to me. > >>>>>>> > > >>>>>>> > What do you think about this? > >>>>>>> > > >>>>>>> > - Eugene > >>>>>>> > > >>>>>>> > > >>>>>>> > On Wed, Feb 8, 2023 at 1:01 AM Julien Le Dem > <jul...@astronomer.io.invalid> wrote: > >>>>>>> >> > >>>>>>> >> Hello Michał, > >>>>>>> >> Thank you for your input. > >>>>>>> >> I would clarify that OpenLineage doesn't make any assumption > about the backend being used to store lineage and is an adapter-like layer. > >>>>>>> >> OpenLineage exists as the spec specifically for that purpose of > avoiding the problem of every lineage consumer having to understand every > lineage producer. > >>>>>>> >> Consumers of lineage want a unified spec consuming lineage from > any data transformation layer like Airflow, Spark, Flink, SQL, Warehouses, > ... > >>>>>>> >> Just like OpenTelemetry allows consuming traces independently > of the technology used, so does OpenLineage for lineage. > >>>>>>> >> Julien > >>>>>>> >> > >>>>>>> >> On Tue, Feb 7, 2023 at 12:48 AM Michał Modras < > michalmod...@google.com> wrote: > >>>>>>> >>> > >>>>>>> >>> Hi everyone, > >>>>>>> >>> > >>>>>>> >>> As Airflow already supports lineage functionality through > pluggable lineage backends, I think OpenLineage and other lineage systems > integration should follow this path. I think more 'native' integration with > OpenLineage (or any other lineage system) in Airflow while maintaining the > generic lineage backend architecture in parallel would make the user > experience less open, troublesome to maintain, and the Airflow architecture > itself more constrained by a logic of a specific system. > >>>>>>> >>> > >>>>>>> >>> I think enriching operators with a generic method exposing > lineage metadata that could be leveraged by lineage backends regardless of > their implementation is a good idea which the Cloud Composer team would > gladly contribute to. I believe the translation of the Airflow metadata > exposed by the operators should be done by lineage backends (or another > adapter-like layer). Tying Airflow operators' development to a specific > lineage system like OpenLineage forces operators' contributors to > understand that system too, which increases both the entry costs and > maintenance costs. I see it as unnecessary coupling. > >>>>>>> >>> > >>>>>>> >>> Best, > >>>>>>> >>> Michal > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> On Tue, Jan 31, 2023 at 7:10 PM Julien Le Dem < > jul...@astronomer.io> wrote: > >>>>>>> >>>> > >>>>>>> >>>> Thank you Eugen, > >>>>>>> >>>> This sounds very aligned with the goals of OpenLineage and I > think this would work well. > >>>>>>> >>>> Here are the sections in the doc that I think address your > points: > >>>>>>> >>>> - generalize lineage metadata extraction as self-method in > each operator, using generic lineage entities > >>>>>>> >>>> See: OpenLineage support in providers. It describes how each > operator exposes its lineage. > >>>>>>> >>>> - implement "adapter"s to convert generated metadata to Data > Lineage format, Open Lineage format, etc. > >>>>>>> >>>> The goal here is each consumer turns from OpenLineage format > to their own internal representation as you are suggesting. > >>>>>>> >>>> In the motivation section, towards the end, I link to a few > examples of data catalogs doing just that. > >>>>>>> >>>> > >>>>>>> >>>> On Tue, Jan 31, 2023 at 8:36 AM Eugen Kosteev < > eu...@kosteev.com> wrote: > >>>>>>> >>>>> > >>>>>>> >>>>> ++ Michal Modras > >>>>>>> >>>>> > >>>>>>> >>>>> On Tue, Jan 31, 2023 at 3:49 PM Eugen Kosteev < > eu...@kosteev.com> wrote: > >>>>>>> >>>>>> > >>>>>>> >>>>>> Cloud Composer recently launched "Data lineage with > Dataplex" feature which effectively means to generate lineage out of > DAG/task executions and export it to Data Lineage (Data Catalog service) > for further analysis. > >>>>>>> >>>>>> > https://cloud.google.com/composer/docs/composer-2/lineage-integration > >>>>>>> >>>>>> > >>>>>>> >>>>>> This feature is as of now in the "Preview" state. > >>>>>>> >>>>>> The current implementation uses built-in "Airflow lineage > backend" feature and methods to extract lineage metadata on task post > execution events. > >>>>>>> >>>>>> > >>>>>>> >>>>>> The general idea was to contribute this to the Airflow > community in a form: > >>>>>>> >>>>>> - generalize lineage metadata extraction as self-method in > each operator, using generic lineage entities > >>>>>>> >>>>>> - implement "adapter"s to convert generated metadata to > Data Lineage format, Open Lineage format, etc. > >>>>>>> >>>>>> > >>>>>>> >>>>>> Adoption of "Airflow OpenLineage" for Composer would mean > to introduce an additional layer of converting from OpenLineage format to > Data Lineage (Data Catalog/Dataplex) format. But this is definitely a > possibility. > >>>>>>> >>>>>> > >>>>>>> >>>>>> On Tue, Jan 31, 2023 at 12:53 AM Julien Le Dem > <jul...@astronomer.io.invalid> wrote: > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> Thank you very much for your input Jarek. > >>>>>>> >>>>>>> I am responding in the comments and adding to the doc > accordingly. > >>>>>>> >>>>>>> I would also love to hear from more stakeholders. > >>>>>>> >>>>>>> Thanks to all who provided feedback so far. > >>>>>>> >>>>>>> Julien > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> On Fri, Jan 27, 2023 at 12:57 AM Jarek Potiuk < > ja...@potiuk.com> wrote: > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> General comment from my side: I think Open Lineage is > (and should be > >>>>>>> >>>>>>>> even more) a feature of Airflow that expands Airflow's > capabilities > >>>>>>> >>>>>>>> greatly and opens up the direction we've been all working > on - Airflow > >>>>>>> >>>>>>>> as a Platform. > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> I think closely integrating it with Open-Lineage goes the > same > >>>>>>> >>>>>>>> direction (also mentioned in the doc) as Open Telemetry > goes, where we > >>>>>>> >>>>>>>> might decide to support certain standards in order to > expand > >>>>>>> >>>>>>>> capabilities of Airflow-as-a-platform and allows to > plug-in multiple > >>>>>>> >>>>>>>> external solutions that would use the standard API. After > Open-Lineage > >>>>>>> >>>>>>>> graduated recently to LFAI&Data foundation (I've been > watching this > >>>>>>> >>>>>>>> happening from far), it is I think the perfect candidate > for Airflow > >>>>>>> >>>>>>>> to incorporate it. I hope this will help all the players > to make use > >>>>>>> >>>>>>>> of the extra work necessary by the community to make it > "officially > >>>>>>> >>>>>>>> supported". I think we have to also get some feedback > from the big > >>>>>>> >>>>>>>> stakeholders in Airflow - because one thing is to have > such a > >>>>>>> >>>>>>>> capability, and another is to get it used in all the ways > Airflow is > >>>>>>> >>>>>>>> used - not only by on-premise/self-hosted users (which is > obviously a > >>>>>>> >>>>>>>> huge driving factor) but also everywhere where Airflow is > exposed by > >>>>>>> >>>>>>>> others - Astronomer is obviously on-board. we see some > warm words from > >>>>>>> >>>>>>>> Amazon (mentioned by Julian), I would love to hear > whether the > >>>>>>> >>>>>>>> Composer team at Google would be on board in using the > open-lineage > >>>>>>> >>>>>>>> information exposed this way in their Data Catalog (and > likely more) > >>>>>>> >>>>>>>> offering. We have Amundsen and others and possibly other > stakeholders > >>>>>>> >>>>>>>> might want to say something. > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> There is - undoubtedly - an extra effort involved in > implementing and > >>>>>>> >>>>>>>> keeping it running smoothly (as Julian mentioned, that is > the main > >>>>>>> >>>>>>>> reason why the Open Lineage community would like to make > the > >>>>>>> >>>>>>>> integration part of Airflow. But by being smart and > integrating it in > >>>>>>> >>>>>>>> the way that will allow to plug-it-in into our CI, > verification > >>>>>>> >>>>>>>> process and making some very clear expectations about > what it means > >>>>>>> >>>>>>>> for contributors to Airflow to get it running, we can > make some > >>>>>>> >>>>>>>> initial investment in making it happen and minimise > on-going cost, > >>>>>>> >>>>>>>> while maximising the gain. > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> And looking at all the above - I am super happy to help > with all that > >>>>>>> >>>>>>>> to make this easy to "swallow" and integrate well, even > if it will > >>>>>>> >>>>>>>> take an extra effort, especially that we will have > experts from Open > >>>>>>> >>>>>>>> Lineage who worked with both Airflow and Open Lineage > being the core > >>>>>>> >>>>>>>> part of the effort. I am actually super excited - this > might be the > >>>>>>> >>>>>>>> next-big-thing for Airflow to strengthen its position as > an > >>>>>>> >>>>>>>> indispensable component of "even more modern data stack". > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> I made my initial comments in the doc, and am looking > forward to > >>>>>>> >>>>>>>> making it happen :). > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> J. > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> On Fri, Jan 27, 2023 at 2:20 AM Julien Le Dem > >>>>>>> >>>>>>>> <jul...@astronomer.io.invalid> wrote: > >>>>>>> >>>>>>>> > > >>>>>>> >>>>>>>> > Dear Airflow Community, > >>>>>>> >>>>>>>> > I have been working on a proposal to bring an > OpenLineage provider to Airflow. > >>>>>>> >>>>>>>> > I am looking for feedback with the goal to post an > official AIP. > >>>>>>> >>>>>>>> > Please feel free to comment in the doc above. > >>>>>>> >>>>>>>> > Thank you, > >>>>>>> >>>>>>>> > Julien (OpenLineage project lead) > >>>>>>> >>>>>>>> > > >>>>>>> >>>>>>>> > For convenience, here is the rationale from the doc: > >>>>>>> >>>>>>>> > > >>>>>>> >>>>>>>> > Operational lineage collection is a common need to > understand dependencies between data pipelines and track end-to-end > provenance of data. It enables many use cases from ensuring reliable > delivery of data through observability to compliance and cost management. > >>>>>>> >>>>>>>> > > >>>>>>> >>>>>>>> > Publishing operational lineage is a core Airflow > capability to enable troubleshooting and governance. > >>>>>>> >>>>>>>> > > >>>>>>> >>>>>>>> > OpenLineage is a project part of the LFAI&Data > foundation that provides a spec standardizing operational lineage > collection and sharing across the data ecosystem. If it provides plugins > for popular open source projects, its intent is very similar to > OpenTelemetry (also under the Linux Foundation umbrella): to remain a spec > for lineage exchange that projects - open source or proprietary - implement. > >>>>>>> >>>>>>>> > > >>>>>>> >>>>>>>> > Built-in OpenLineage support in Airflow will make it > easier and more reliable for Airflow users to publish their operational > lineage through the OpenLineage ecosystem. > >>>>>>> >>>>>>>> > > >>>>>>> >>>>>>>> > The current external plugin maintained in the > OpenLineage project depends on Airflow and operators internals and gets > broken when changes are made on those. Having a built-in integration > ensures a better first class support to expose lineage that gets tested > alongside other changes and therefore is more stable. > >>>>>>> >>>>>> > >>>>>>> >>>>>> > >>>>>>> >>>>>> > >>>>>>> >>>>>> -- > >>>>>>> >>>>>> Eugene > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> -- > >>>>>>> >>>>> Eugene > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Eugene >
