I like the idea of having a common.compat provider too. (common.util was actually kinda confusing)
* when providers get >= airflow 2.10 - we change them to import from > `airflow.openlineage` rather than from "airflow.providers.common.compat". > I am not so sure why this is the case. Can you elaborate? Thanks & Regards, Amogh Desai On Mon, Jun 10, 2024 at 4:15 PM Jakub Dardziński <kuba0...@gmail.com> wrote: > As the author of https://github.com/apache/airflow/pull/39530 I love the > idea. > > * when providers get >= airflow 2.10 - we change them to import from > > `airflow.openlineage` rather than from "airflow.providers.common.compat". > > > What's the reasoning behind that? How would Airflow core release impact > providers dependencies? > > pon., 10 cze 2024 o 11:08 Maciej Obuchowski <obuchowski.mac...@gmail.com> > napisał(a): > > > I think it's a good solution. > > The only known problem with that idea is that the common code has to live > > "forever" - as long as someone can use the older providers (or older > > Airflow version). > > The solution would be to introduce some explicit deprecation or > versioning > > for provider dependencies - but that's not really possible due to lack of > > constraints > > for optional dependencies. > > > > sob., 8 cze 2024 o 22:00 Jarek Potiuk <ja...@potiuk.com> napisał(a): > > > > > I have an idea about that one, and probably that one will fulfill the > > > "polyfill" approach discussed earlier. > > > > > > I think we should not name the provider "common.util" but > > "common.compat" - > > > because all the code that we need to put there is really about keeping > > > compatibility. > > > > > > For example look here https://github.com/apache/airflow/pull/39530 > > > > > > We have a need to have a "compatibility" code somewhere that a number > of > > > providers could use in case we want to keep some backwards > compatibility. > > > > > > So having a "common.compat" provider would likely nicely full-fill the > > > polyfill approach - It should only contain the code that we aim to keep > > > backwards compatibility > > > > > > Example for https://github.com/apache/airflow/pull/39530 > > > > > > * we add the complex compatibility code (see > > > https://github.com/apache/airflow/pull/39530#issuecomment-2145670785) > in > > > the "common.compat" provider - and to airflow.openlineage in this case > > > * we import it from there in all providers that need it (this will > > > automatically add dependency) > > > * when providers get >= airflow 2.10 - we change them to import from > > > `airflow.openlineage` rather than from > "airflow.providers.common.compat". > > > > > > We could apply similar approach for other "compatibility" code > > > > > > J. > > > > > > > > > > > > > > > On Thu, Apr 11, 2024 at 10:22 AM Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > > Any other ideas or suggestions here? Can someone explain how the > > > > "polypill" approach would look like, maybe? How do we imagine this > > > working? > > > > > > > > Just to continue this discussion - another example. > > > > > > > > Small thing that David wanted to add for changes in some sql > providers: > > > > > > > > @contextmanager > > > > def suppress_and_warn(*exceptions: type[BaseException]): > > > > """Context manager that suppresses the given exceptions and logs > a > > > > warning message.""" > > > > try: > > > > yield > > > > except exceptions as e: > > > > warnings.warn(f"Exception suppressed: > > > > {e}\n{traceback.format_exc()}", category=UserWarning) > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/pull/38707/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R115 > > > > > > > > This is a small thing - but adding it in `airflow` is problematic - > > > > because it will only be released in 1.10, so we cannot use it in > > > providers > > > > if we do. > > > > Currently - since it is used in sql providers, I suggested using > > > > `common.sql` for that code (and add >= 1.12 for common-sql-providers > > for > > > > those providers that use it). And I will write a separate email > about a > > > > proposed versioning approach there. > > > > > > > > Do we have a good proposal on how we can solve similar things in the > > > > future? > > > > Do we want it at all? It has some challenges - yes it DRY's the code > > but > > > > it also introduces coupling. > > > > > > > > J. > > > > > > > > > > > > On Sun, Mar 10, 2024 at 6:21 PM Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > > >> Coming back to it - what about the "polypill" :)? What's different > vs > > > the > > > >> "common.sql" way of doing it ? How do we think it can work ? > > > >> > > > >> On Thu, Feb 22, 2024 at 1:58 PM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > >> > > > >>> > The symbolic link approach seems to disregard all the external > > > >>> providers, unless I misunderstand it. > > > >>> > > > >>> Not really. It just does not make it easy for the external > providers > > to > > > >>> use it "fast". They can still - if they want to just manually copy > > > those > > > >>> utils from the latest version of Airflow if they want to use it. > > > Almost by > > > >>> definition, those will be small, independent modules that can be > > simply > > > >>> copied as needed by whoever releases external providers - and they > > are > > > also > > > >>> free to copy any older version if they want. That is a nice feature > > > that > > > >>> makes them fully decoupled from the version of Airflow they are > > > installed > > > >>> in (same as community providers). Or - if they want they can just > > > import > > > >>> them from "airflow.provider_utils" - but then they have to add >= > > > Airflow > > > >>> 2.9 if that util appeared in Airflow 2.9 (which is the main reason > we > > > want > > > >>> to use symbolic links - because due to our policies and promises, > we > > > do not > > > >>> want community providers to depend on latest version of Airflow in > > vast > > > >>> majority of cases. > > > >>> > > > >>> So this approach is also fully usable by external providers, but it > > > >>> requires some manual effort to copy the modules to their providers. > > > >>> > > > >>> > I like the polypill idea. A backport provider that brings new > > > >>> interfaces to providers without the actual functionalities. > > > >>> > > > >>> I would love to hear more about this, I think the "common.util" was > > > >>> exactly the kind of polyfill approach (with its own versioning > > > >>> complexities) but maybe I do not understand how such a polyfill > > > provider > > > >>> would work. Say we want to add a new "urlparse" method usable for > all > > > >>> providers. Could you explain how it would work - say: > > > >>> > > > >>> * we add "urlparse" in Airflow 2.9 > > > >>> * some provider wants to use it in Airflow 2.7 > > > >>> > > > >>> What providers, with what code/interfaces we would have to release > in > > > >>> this case and what dependencies such providers that want to use it > > > (both > > > >>> community and Airflow should have)? I **think** that would mean > > > exactly the > > > >>> "common.<something>" approach we already have with "io" and "sql", > > but > > > >>> maybe I do not understand it :) > > > >>> > > > >>> On Thu, Feb 22, 2024 at 1:45 PM Tzu-ping Chung > > > <t...@astronomer.io.invalid> > > > >>> wrote: > > > >>> > > > >>>> I like the polypill idea. A backport provider that brings new > > > >>>> interfaces to providers without the actual functionalities. > > > >>>> > > > >>>> > > > >>>> > On 22 Feb 2024, at 20:41, Maciej Obuchowski < > > mobuchow...@apache.org > > > > > > > >>>> wrote: > > > >>>> > > > > >>>> >> That's why I generally do > > > >>>> > not like the "util" approach because common packaging introduces > > > >>>> > unnecessary coupling (you have to upgrade independent utils > > > together). > > > >>>> > > > > >>>> > From my experience with releasing OpenLineage where we do things > > > >>>> similarly: > > > >>>> > I think that's the advantage of it, but only _if_ you can > release > > > >>>> those > > > >>>> > together. > > > >>>> > With "build-in" providers it makes sense, but could be > burdensome > > if > > > >>>> > "external" > > > >>>> > ones would depend on that functionality. > > > >>>> > > > > >>>> >> I know it's not been the original idea behind providers, but - > > > after > > > >>>> > testing common.sql and now also having common.io, seems like > more > > > >>>> and more > > > >>>> > we would like to extract some common code that we would like > > > >>>> providers to > > > >>>> > use, but we refrain from it, because it will only be actually > > > usable 6 > > > >>>> > months after we introduce some common code. > > > >>>> > > > > >>>> > So, maybe better approach would be to introduce the > functionality > > > into > > > >>>> > core, > > > >>>> > and use common.X provider as "polyfill" (to borrow JS > > nomenclature) > > > >>>> > to make sure providers could use that functionality now, where > > > >>>> external > > > >>>> > ones could depend on the Airflow ones? > > > >>>> > > > > >>>> > The symbolic link approach seems to disregard all the external > > > >>>> providers, > > > >>>> > unless > > > >>>> > I misunderstand it. > > > >>>> > > > > >>>> > czw., 22 lut 2024 o 13:28 Jarek Potiuk <ja...@potiuk.com> > > > napisał(a): > > > >>>> > > > > >>>> >>> Ideally utilities for each purpose (parsing URI, reading > Object > > > >>>> Storage, > > > >>>> >> reading SQL, etc.) should each have its own utility package, so > > > they > > > >>>> can be > > > >>>> >> released independently without dependency problems popping up > if > > we > > > >>>> need to > > > >>>> >> break compatibility to one purpose. But more providers are > > > >>>> exponentially > > > >>>> >> more difficult to maintain, so I’d settle for one utility > > provider > > > >>>> for now > > > >>>> >> and split further if needed in the future. > > > >>>> >> > > > >>>> >> Very much agree with this general statement. That's why I > > generally > > > >>>> do > > > >>>> >> not like the "util" approach because common packaging > introduces > > > >>>> >> unnecessary coupling (you have to upgrade independent utils > > > >>>> together). And > > > >>>> >> when we have a common set of things that seem to make sense to > be > > > >>>> released > > > >>>> >> together when upgraded we should package them together in > > > >>>> >> "common.<something concrete" (like we have with common.io and > > > >>>> common.sql). > > > >>>> >> > > > >>>> >> However - in this case, I think what Jens proposed (and I am > > happy > > > >>>> to try > > > >>>> >> as well) is to attempt to use symbolic links - i.e. add the > code > > in > > > >>>> >> `airflow.util` but then create a symbolic link in the provider. > > I > > > >>>> tested > > > >>>> >> it yesterday and it works as expected - i.e. such symbolic link > > is > > > >>>> >> dereferenced and the provider package contains the python file, > > not > > > >>>> >> symbolic link. That seems like a much more lightweight approach > > > that > > > >>>> will > > > >>>> >> serve the purpose of "common.util" much better. The only thing > we > > > >>>> will have > > > >>>> >> to take care of (and we can add it once the POC is successful) > is > > > to > > > >>>> add > > > >>>> >> some pre-commit protection that those kind of symbolically > linked > > > >>>> util > > > >>>> >> modules are imported in providers, from inside of those > provider, > > > >>>> not from > > > >>>> >> airlfow, and make sure they are "standalone" (i.e. - as you > > > >>>> mentioned - not > > > >>>> >> depend on anything in airflow code). We could create a new > > package > > > >>>> for that > > > >>>> >> in airlfow > > > >>>> >> "airlfow.provider_utils" for example - and make sure (as next > > step) > > > >>>> that > > > >>>> >> anything from that package is never directly imported by any > > > >>>> provider, and > > > >>>> >> whenever provider uses it, it should be symbolic link inside > that > > > >>>> provider. > > > >>>> >> That's all automatable and we can prevent mistakes via > > pre-commit. > > > >>>> >> > > > >>>> >> I think that might lead to a very lightweight approach where we > > > >>>> introduce > > > >>>> >> new common functionality which is immediately reusable in > > providers > > > >>>> without > > > >>>> >> the hassle of taking care about backwards compatibility, and > > > >>>> managing the > > > >>>> >> "common.util" provider. At the expense of a bit complex > > pre-commit > > > >>>> that > > > >>>> >> will guard the usage of it. > > > >>>> >> > > > >>>> >> Seems that it might be the "Eat cake and have it too" way that > > > we've > > > >>>> been > > > >>>> >> looking for. > > > >>>> >> > > > >>>> >> J. > > > >>>> >> > > > >>>> >> On Thu, Feb 22, 2024 at 6:14 AM Tzu-ping Chung > > > >>>> <t...@astronomer.io.invalid> > > > >>>> >> wrote: > > > >>>> >> > > > >>>> >>> It would help in the sense mentioned in previous posts, yes. > But > > > one > > > >>>> >> thing > > > >>>> >>> I want to point out is, for the provider to actually be > helpful, > > > it > > > >>>> must > > > >>>> >> be > > > >>>> >>> treated a bit differently from normal providers, but more > like a > > > >>>> separate > > > >>>> >>> third-party dependency. Specifically, the provider should not > > > have a > > > >>>> >>> dependency to Core Airflow, so it can be released and depended > > on > > > >>>> more > > > >>>> >>> flexibly. > > > >>>> >>> > > > >>>> >>> Ideally utilities for each purpose (parsing URI, reading > Object > > > >>>> Storage, > > > >>>> >>> reading SQL, etc.) should each have its own utility package, > so > > > >>>> they can > > > >>>> >> be > > > >>>> >>> released independently without dependency problems popping up > if > > > we > > > >>>> need > > > >>>> >> to > > > >>>> >>> break compatibility to one purpose. But more providers are > > > >>>> exponentially > > > >>>> >>> more difficult to maintain, so I’d settle for one utility > > provider > > > >>>> for > > > >>>> >> now > > > >>>> >>> and split further if needed in the future. > > > >>>> >>> > > > >>>> >>> TP > > > >>>> >>> > > > >>>> >>> > > > >>>> >>>> On 22 Feb 2024, at 10:10, Scheffler Jens (XC-AS/EAE-ADA-T) < > > > >>>> >>> jens.scheff...@de.bosch.com.INVALID> wrote: > > > >>>> >>>> > > > >>>> >>>> @Uranusjr would this help as a pilot in your AIP-60 code to > > parse > > > >>>> and > > > >>>> >>> validate URIs for datasets? > > > >>>> >>>> > > > >>>> >>>> Mit freundlichen Grüßen / Best regards > > > >>>> >>>> > > > >>>> >>>> Jens Scheffler > > > >>>> >>>> > > > >>>> >>>> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > > > >>>> >>>> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 > > > >>>> Stuttgart-Vaihingen | > > > >>>> >>> GERMANY | www.bosch.com > > > >>>> >>>> Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > > >>>> >>> jens.scheff...@de.bosch.com > > > >>>> >>>> > > > >>>> >>>> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB > > > 14000; > > > >>>> >>>> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > > >>>> >>>> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, > > Dr. > > > >>>> Markus > > > >>>> >>> Forschner, > > > >>>> >>>> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja > > > Rückert > > > >>>> >>>> > > > >>>> >>>> -----Original Message----- > > > >>>> >>>> From: Jarek Potiuk <ja...@potiuk.com> > > > >>>> >>>> Sent: Donnerstag, 22. Februar 2024 00:53 > > > >>>> >>>> To: dev@airflow.apache.org > > > >>>> >>>> Subject: Re: [DISCUSS] Common.util provider? > > > >>>> >>>> > > > >>>> >>>> Yep. It could work with symbolic links. Tested it and with > > flit - > > > >>>> both > > > >>>> >>> wheel and sdist packaged code such symbolically linked file is > > > >>>> >> dereferenced > > > >>>> >>> and copy of the file is added there. It could be a nice way of > > > >>>> doing it. > > > >>>> >>>> > > > >>>> >>>> Maybe then worth trying next time if someone has a need? > > > >>>> >>>> > > > >>>> >>>> J > > > >>>> >>>> > > > >>>> >>>> On Thu, Feb 22, 2024 at 12:39 AM Scheffler Jens > > > (XC-AS/EAE-ADA-T) < > > > >>>> >>> jens.scheff...@de.bosch.com.invalid> wrote: > > > >>>> >>>> > > > >>>> >>>>>>>> As of additional dependency complexity between providers > > > >>>> actually > > > >>>> >>>>>>>> the > > > >>>> >>>>> additional dependency I think creates more problems than the > > > >>>> benefit… > > > >>>> >>>>> would be cool if there would be an option to „inline“ common > > > code > > > >>>> from > > > >>>> >>>>> a single place but keep individual providers fully > > independent… > > > >>>> >>>>> > > > >>>> >>>>>> Well, we already do a lot of inlining, so if we think we > > > should > > > >>>> do > > > >>>> >>>>>> more, > > > >>>> >>>>> we have mechanisms for that. We have pre-commits and > release > > > >>>> commands > > > >>>> >>>>> that do a lot of that. Pre commits are inlining scripts in > > > >>>> >>>>> Dockerfiles, shortening PyPI readme . The providers > > __init__.py > > > >>>> files > > > >>>> >>>>> and changelogs and index documentation .rst (partially) are > > > >>>> generated > > > >>>> >>>>> at release documentation preparation time, pyproject.toml > for > > > >>>> >>>>> providers are generated from common templates at package > > > building > > > >>>> time > > > >>>> >>>>> and so on and so on :). So we can do more of that and > generate > > > >>>> common > > > >>>> >>>>> code, it's just a matter of adding pre-commits or breeze > > > scripts. > > > >>>> But > > > >>>> >>>>> again "can't have and eat cake" - this has the drawback that > > > >>>> there are > > > >>>> >>>>> extra steps involved and even if it's automated it does add > > > >>>> friction > > > >>>> >>>>> when you have to regenerate the code every time you change > it > > > and > > > >>>> when > > > >>>> >>>>> you change it in another place than where you use it. > > > >>>> >>>>> > > > >>>> >>>>> Yes, also thought a moment about pre-commit. I#d be okay if > we > > > >>>> really > > > >>>> >>>>> in-line and have a pre-commit aligning the redundancy of > > python > > > in > > > >>>> >>> folders. > > > >>>> >>>>> Might need to be an opt-in if only 10 of 85 providers are > > using > > > >>>> common > > > >>>> >>>>> stuff and if we change a common line we probably do not need > > to > > > >>>> affect > > > >>>> >>>>> all providers. > > > >>>> >>>>> > > > >>>> >>>>> As long as no Windows users trying to code for airflow (do > we > > > >>>> need to > > > >>>> >>>>> consider?) would it also work to use symlinks? Git can cope > > with > > > >>>> this, > > > >>>> >>>>> I don't know if the python toolchain can de-reference a copy > > and > > > >>>> are > > > >>>> >>>>> not packaging a symlink? Would be worth a test... would save > > the > > > >>>> >>>>> pre-commit and we even could selectively include common bla > > into > > > >>>> >>>>> providers as needed :-D > > > >>>> >>>>> > > > >>>> >>>>> Mit freundlichen Grüßen / Best regards > > > >>>> >>>>> > > > >>>> >>>>> Jens Scheffler > > > >>>> >>>>> > > > >>>> >>>>> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert Bosch > > > GmbH > > > >>>> | > > > >>>> >>>>> Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | > > > >>>> >>>>> www.bosch.com Tel. +49 711 811-91508 | Mobil +49 160 > > 90417410 | > > > >>>> >>>>> jens.scheff...@de.bosch.com > > > >>>> >>>>> > > > >>>> >>>>> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB > > > >>>> 14000; > > > >>>> >>>>> Aufsichtsratsvorsitzender: Prof. Dr. Stefan > Asenkerschbaumer; > > > >>>> >>>>> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, > > Dr. > > > >>>> >>>>> Markus Forschner, Stefan Grosch, Dr. Markus Heyn, Dr. Frank > > > >>>> Meyer, Dr. > > > >>>> >>>>> Tanja Rückert > > > >>>> >>>>> > > > >>>> >>>>> -----Original Message----- > > > >>>> >>>>> From: Jarek Potiuk <ja...@potiuk.com> > > > >>>> >>>>> Sent: Mittwoch, 21. Februar 2024 21:18 > > > >>>> >>>>> To: dev@airflow.apache.org > > > >>>> >>>>> Subject: Re: [DISCUSS] Common.util provider? > > > >>>> >>>>> > > > >>>> >>>>>> if we have a common piece then we are locking all depending > > > >>>> >>>>>> providers > > > >>>> >>>>> (potentially) together if common code changes > > > >>>> >>>>> > > > >>>> >>>>> Yes, coupling in this case is the drawback of this solution. > > You > > > >>>> can't > > > >>>> >>>>> have cake and eat it too and in this case you trade DRY with > > > >>>> coupling. > > > >>>> >>>>> > > > >>>> >>>>>> As of additional dependency complexity between providers > > > actually > > > >>>> >>>>>> the > > > >>>> >>>>> additional dependency I think creates more problems than the > > > >>>> benefit… > > > >>>> >>>>> would be cool if there would be an option to „inline“ common > > > code > > > >>>> from > > > >>>> >>>>> a single place but keep individual providers fully > > independent… > > > >>>> >>>>> > > > >>>> >>>>> Well, we already do a lot of inlining, so if we think we > > should > > > >>>> do > > > >>>> >>>>> more, we have mechanisms for that. We have pre-commits and > > > >>>> release > > > >>>> >>>>> commands that do a lot of that. Pre commits are inlining > > scripts > > > >>>> in > > > >>>> >>>>> Dockerfiles, shortening PyPI readme . The providers > > __init__.py > > > >>>> files > > > >>>> >>>>> and changelogs and index documentation .rst (partially) are > > > >>>> generated > > > >>>> >>>>> at release documentation preparation time, pyproject.toml > for > > > >>>> >>>>> providers are generated from common templates at package > > > building > > > >>>> time > > > >>>> >>>>> and so on and so on :). So we can do more of that and > generate > > > >>>> common > > > >>>> >>>>> code, it's just a matter of adding pre-commits or breeze > > > scripts. > > > >>>> But > > > >>>> >>>>> again "can't have and eat cake" - this has the drawback that > > > >>>> there are > > > >>>> >>>>> extra steps involved and even if it's automated it does add > > > >>>> friction > > > >>>> >>>>> when you have to regenerate the code every time you change > it > > > and > > > >>>> when > > > >>>> >>>>> you change it in another place than where you use it. > > > >>>> >>>>> > > > >>>> >>>>> J. > > > >>>> >>>>> > > > >>>> >>>>> On Wed, Feb 21, 2024 at 9:02 PM Scheffler Jens > > > (XC-AS/EAE-ADA-T) < > > > >>>> >>>>> jens.scheff...@de.bosch.com.invalid> wrote: > > > >>>> >>>>> > > > >>>> >>>>>> Hi Jarek, > > > >>>> >>>>>> > > > >>>> >>>>>> At reviewing the PR from uranusjr for AIP-60 I also had the > > > >>>> feeling > > > >>>> >>>>>> that a lot of very similar code is repeated in all the > > > providers. > > > >>>> >>>>>> But during review yesterday I dropped the ides because if > we > > > >>>> have a > > > >>>> >>>>>> common piece then we are locking all depending providers > > > >>>> >>>>>> (potentially) together if common code changes. > > > >>>> >>>>>> As of additional dependency complexity between providers > > > actually > > > >>>> >>>>>> the additional dependency I think creates more prblems than > > the > > > >>>> >>>>>> benefit… would be cool if tehere would be an option to > > „inline“ > > > >>>> >>>>>> common code from a single place but keep individual > providers > > > >>>> fully > > > >>>> >>>>>> independent… > > > >>>> >>>>>> > > > >>>> >>>>>> Jens > > > >>>> >>>>>> > > > >>>> >>>>>> Sent from Outlook for > > > >>>> >>>>>> iOS< > > > >>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F% > > > >>>> >>>>>> 2F > > > >>>> >>>>>> aka.ms%2Fo0ukef&data=05%7C02%7CJens.Scheffler% > 40de.bosch.com > > > >>>> %7C98c88 > > > >>>> >>>>>> 97 > > > >>>> >>>>>> > > > >>>> > 195d944d483ab08dc331a49bb%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0 > > > >>>> >>>>>> %7 > > > >>>> >>>>>> > > > >>>> > C638441435197193656%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ > > > >>>> >>>>>> Ij > > > >>>> >>>>>> > > > >>>> > oiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=n6gk9fNn > > > >>>> >>>>>> WB SJOPYEgJ9WbriZ3H4id3RhLr16SguOuFA%3D&reserved=0> > > > >>>> >>>>>> ________________________________ > > > >>>> >>>>>> From: Jarek Potiuk <ja...@potiuk.com> > > > >>>> >>>>>> Sent: Wednesday, February 21, 2024 5:42:20 PM > > > >>>> >>>>>> To: dev@airflow.apache.org <dev@airflow.apache.org> > > > >>>> >>>>>> Subject: [DISCUSS] Common.util provider? > > > >>>> >>>>>> > > > >>>> >>>>>> Hello everyone, > > > >>>> >>>>>> > > > >>>> >>>>>> How do we feel about introducing a common.util provider? > > > >>>> >>>>>> > > > >>>> >>>>>> I know it's not been the original idea behind providers, > but > > - > > > >>>> after > > > >>>> >>>>>> testing common.sql and now also having common.io, seems > like > > > >>>> more > > > >>>> >>>>>> and more we would like to extract some common code that we > > > would > > > >>>> >>>>>> like providers to use, but we refrain from it, because it > > will > > > >>>> only > > > >>>> >>>>>> be actually usable 6 months after we introduce some common > > > code. > > > >>>> >>>>>> > > > >>>> >>>>>> However, if we introduce common.util, this problem is > > generally > > > >>>> gone > > > >>>> >>>>>> - at the expense of more complex maintenance and > > cross-provider > > > >>>> >>>>> dependencies. > > > >>>> >>>>>> We should be able to add a common util method and use it > in a > > > >>>> >>>>>> provider at the same time. > > > >>>> >>>>>> > > > >>>> >>>>>> Think Amazon provider using a new feature released in > > > common.util > > > >>>> >>>>>>> =1.2.0 and google provider >= 1.1.0. All manageable and we > > do > > > it > > > >>>> >>>>>> already for common.sql. We know how to do it, we know what > to > > > >>>> avoid, > > > >>>> >>>>>> we know we cannot introduce backwards-incompatible changes, > > so > > > we > > > >>>> >>>>>> have to be very clear what is and what is not a public API > > > >>>> there, We > > > >>>> >>>>>> could rather easily add tests to prevent such > > > >>>> backwards-incompatible > > > >>>> >>>>>> changes. We even have a solution for chicken-egg providers > > > where > > > >>>> we > > > >>>> >>>>>> need to release two providers at the same time if they > depend > > > on > > > >>>> >>>>>> each other. Generally speaking it's quite workable but > adds a > > > >>>> bit of > > > >>>> >>> overhead. > > > >>>> >>>>>> > > > >>>> >>>>>> Examples that we could implement as "common.util": > > > >>>> >>>>>> > > > >>>> >>>>>> - common versioning check with cache - where multiple > > providers > > > >>>> >>>>>> could reuse "do we have pendulum 2" > > > >>>> >>>>>> - more complex - some date management features (we have a > few > > > >>>> like > > > >>>> >>>>>> date_ranges/round_time). But there are many more. > > > >>>> >>>>>> > > > >>>> >>>>>> I generally do not love the common "util" approach. It has > a > > > >>>> >>>>>> tendency to become a bag of everything over time. but if we > > > >>>> limit it > > > >>>> >>>>>> to a set of small, fully decoupled modules where each > module > > is > > > >>>> >>>>>> independent - it's OK. And we already have it in > > "airflow.util" > > > >>>> and > > > >>>> >>>>>> we seem to be > > > >>>> >>>>> doing well. > > > >>>> >>>>>> > > > >>>> >>>>>> WDYT? Is it worth it ? > > > >>>> >>>>>> > > > >>>> >>>>>> J. > > > >>>> >>>>>> > > > >>>> >>>>> > > > >>>> >>>> > > > >>>> >>>> > > > >>>> > > --------------------------------------------------------------------- > > > >>>> >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >>>> >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > > >>>> >>> > > > >>>> >>> > > > >>>> >> > > > >>>> > > > >>>> > > > >>>> > > --------------------------------------------------------------------- > > > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > > >>>> > > > >>>> > > > > > >
