> The only known problem with that idea is that the common code has to live "forever" - as long as someone can use the older providers (or older Airflow version).
Here Jakub is right - it's only 'older providers" not "older airlfow" - the provider will have "apache-airflow>=2.7.0" so older airflow version is not a problem. It is for older providers version. But. I think this back-compat code should be essentially "frozen" and we should (we can automate it) make sure that when we bump min airflow version, we stop using that old code. Having the old compatibility code laying in a separate provider is a very little issue comparing to having the same code copied across multiple providers for 2-3 years, like it would be in case of https://github.com/apache/airflow/pull/39530 Does anyone have something against it? I think it very much fulfills the polyfill idea (unless someone can come up with a better approach). I will start a LAZY CONSENSUS in a few days if I do not hear more about it :) J, On Mon, Jun 10, 2024 at 4:06 PM Amogh Desai <amoghdesai....@gmail.com> wrote: > I like the idea of having a common.compat provider too. (common.util was > actually kinda confusing) > > * when providers get >= airflow 2.10 - we change them to import from > > `airflow.openlineage` rather than from "airflow.providers.common.compat". > > > I am not so sure why this is the case. Can you elaborate? > > > Thanks & Regards, > Amogh Desai > > > On Mon, Jun 10, 2024 at 4:15 PM Jakub Dardziński <kuba0...@gmail.com> > wrote: > > > As the author of https://github.com/apache/airflow/pull/39530 I love the > > idea. > > > > * when providers get >= airflow 2.10 - we change them to import from > > > `airflow.openlineage` rather than from > "airflow.providers.common.compat". > > > > > What's the reasoning behind that? How would Airflow core release impact > > providers dependencies? > > > > pon., 10 cze 2024 o 11:08 Maciej Obuchowski <obuchowski.mac...@gmail.com > > > > napisał(a): > > > > > I think it's a good solution. > > > The only known problem with that idea is that the common code has to > live > > > "forever" - as long as someone can use the older providers (or older > > > Airflow version). > > > The solution would be to introduce some explicit deprecation or > > versioning > > > for provider dependencies - but that's not really possible due to lack > of > > > constraints > > > for optional dependencies. > > > > > > sob., 8 cze 2024 o 22:00 Jarek Potiuk <ja...@potiuk.com> napisał(a): > > > > > > > I have an idea about that one, and probably that one will fulfill the > > > > "polyfill" approach discussed earlier. > > > > > > > > I think we should not name the provider "common.util" but > > > "common.compat" - > > > > because all the code that we need to put there is really about > keeping > > > > compatibility. > > > > > > > > For example look here https://github.com/apache/airflow/pull/39530 > > > > > > > > We have a need to have a "compatibility" code somewhere that a number > > of > > > > providers could use in case we want to keep some backwards > > compatibility. > > > > > > > > So having a "common.compat" provider would likely nicely full-fill > the > > > > polyfill approach - It should only contain the code that we aim to > keep > > > > backwards compatibility > > > > > > > > Example for https://github.com/apache/airflow/pull/39530 > > > > > > > > * we add the complex compatibility code (see > > > > https://github.com/apache/airflow/pull/39530#issuecomment-2145670785 > ) > > in > > > > the "common.compat" provider - and to airflow.openlineage in this > case > > > > * we import it from there in all providers that need it (this will > > > > automatically add dependency) > > > > * when providers get >= airflow 2.10 - we change them to import from > > > > `airflow.openlineage` rather than from > > "airflow.providers.common.compat". > > > > > > > > We could apply similar approach for other "compatibility" code > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > On Thu, Apr 11, 2024 at 10:22 AM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > > > > > > > Any other ideas or suggestions here? Can someone explain how the > > > > > "polypill" approach would look like, maybe? How do we imagine this > > > > working? > > > > > > > > > > Just to continue this discussion - another example. > > > > > > > > > > Small thing that David wanted to add for changes in some sql > > providers: > > > > > > > > > > @contextmanager > > > > > def suppress_and_warn(*exceptions: type[BaseException]): > > > > > """Context manager that suppresses the given exceptions and > logs > > a > > > > > warning message.""" > > > > > try: > > > > > yield > > > > > except exceptions as e: > > > > > warnings.warn(f"Exception suppressed: > > > > > {e}\n{traceback.format_exc()}", category=UserWarning) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/pull/38707/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R115 > > > > > > > > > > This is a small thing - but adding it in `airflow` is problematic - > > > > > because it will only be released in 1.10, so we cannot use it in > > > > providers > > > > > if we do. > > > > > Currently - since it is used in sql providers, I suggested using > > > > > `common.sql` for that code (and add >= 1.12 for > common-sql-providers > > > for > > > > > those providers that use it). And I will write a separate email > > about a > > > > > proposed versioning approach there. > > > > > > > > > > Do we have a good proposal on how we can solve similar things in > the > > > > > future? > > > > > Do we want it at all? It has some challenges - yes it DRY's the > code > > > but > > > > > it also introduces coupling. > > > > > > > > > > J. > > > > > > > > > > > > > > > On Sun, Mar 10, 2024 at 6:21 PM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > > > > > > > >> Coming back to it - what about the "polypill" :)? What's different > > vs > > > > the > > > > >> "common.sql" way of doing it ? How do we think it can work ? > > > > >> > > > > >> On Thu, Feb 22, 2024 at 1:58 PM Jarek Potiuk <ja...@potiuk.com> > > > wrote: > > > > >> > > > > >>> > The symbolic link approach seems to disregard all the external > > > > >>> providers, unless I misunderstand it. > > > > >>> > > > > >>> Not really. It just does not make it easy for the external > > providers > > > to > > > > >>> use it "fast". They can still - if they want to just manually > copy > > > > those > > > > >>> utils from the latest version of Airflow if they want to use it. > > > > Almost by > > > > >>> definition, those will be small, independent modules that can be > > > simply > > > > >>> copied as needed by whoever releases external providers - and > they > > > are > > > > also > > > > >>> free to copy any older version if they want. That is a nice > feature > > > > that > > > > >>> makes them fully decoupled from the version of Airflow they are > > > > installed > > > > >>> in (same as community providers). Or - if they want they can just > > > > import > > > > >>> them from "airflow.provider_utils" - but then they have to add >= > > > > Airflow > > > > >>> 2.9 if that util appeared in Airflow 2.9 (which is the main > reason > > we > > > > want > > > > >>> to use symbolic links - because due to our policies and promises, > > we > > > > do not > > > > >>> want community providers to depend on latest version of Airflow > in > > > vast > > > > >>> majority of cases. > > > > >>> > > > > >>> So this approach is also fully usable by external providers, but > it > > > > >>> requires some manual effort to copy the modules to their > providers. > > > > >>> > > > > >>> > I like the polypill idea. A backport provider that brings new > > > > >>> interfaces to providers without the actual functionalities. > > > > >>> > > > > >>> I would love to hear more about this, I think the "common.util" > was > > > > >>> exactly the kind of polyfill approach (with its own versioning > > > > >>> complexities) but maybe I do not understand how such a polyfill > > > > provider > > > > >>> would work. Say we want to add a new "urlparse" method usable for > > all > > > > >>> providers. Could you explain how it would work - say: > > > > >>> > > > > >>> * we add "urlparse" in Airflow 2.9 > > > > >>> * some provider wants to use it in Airflow 2.7 > > > > >>> > > > > >>> What providers, with what code/interfaces we would have to > release > > in > > > > >>> this case and what dependencies such providers that want to use > it > > > > (both > > > > >>> community and Airflow should have)? I **think** that would mean > > > > exactly the > > > > >>> "common.<something>" approach we already have with "io" and > "sql", > > > but > > > > >>> maybe I do not understand it :) > > > > >>> > > > > >>> On Thu, Feb 22, 2024 at 1:45 PM Tzu-ping Chung > > > > <t...@astronomer.io.invalid> > > > > >>> wrote: > > > > >>> > > > > >>>> I like the polypill idea. A backport provider that brings new > > > > >>>> interfaces to providers without the actual functionalities. > > > > >>>> > > > > >>>> > > > > >>>> > On 22 Feb 2024, at 20:41, Maciej Obuchowski < > > > mobuchow...@apache.org > > > > > > > > > >>>> wrote: > > > > >>>> > > > > > >>>> >> That's why I generally do > > > > >>>> > not like the "util" approach because common packaging > introduces > > > > >>>> > unnecessary coupling (you have to upgrade independent utils > > > > together). > > > > >>>> > > > > > >>>> > From my experience with releasing OpenLineage where we do > things > > > > >>>> similarly: > > > > >>>> > I think that's the advantage of it, but only _if_ you can > > release > > > > >>>> those > > > > >>>> > together. > > > > >>>> > With "build-in" providers it makes sense, but could be > > burdensome > > > if > > > > >>>> > "external" > > > > >>>> > ones would depend on that functionality. > > > > >>>> > > > > > >>>> >> I know it's not been the original idea behind providers, but > - > > > > after > > > > >>>> > testing common.sql and now also having common.io, seems like > > more > > > > >>>> and more > > > > >>>> > we would like to extract some common code that we would like > > > > >>>> providers to > > > > >>>> > use, but we refrain from it, because it will only be actually > > > > usable 6 > > > > >>>> > months after we introduce some common code. > > > > >>>> > > > > > >>>> > So, maybe better approach would be to introduce the > > functionality > > > > into > > > > >>>> > core, > > > > >>>> > and use common.X provider as "polyfill" (to borrow JS > > > nomenclature) > > > > >>>> > to make sure providers could use that functionality now, where > > > > >>>> external > > > > >>>> > ones could depend on the Airflow ones? > > > > >>>> > > > > > >>>> > The symbolic link approach seems to disregard all the external > > > > >>>> providers, > > > > >>>> > unless > > > > >>>> > I misunderstand it. > > > > >>>> > > > > > >>>> > czw., 22 lut 2024 o 13:28 Jarek Potiuk <ja...@potiuk.com> > > > > napisał(a): > > > > >>>> > > > > > >>>> >>> Ideally utilities for each purpose (parsing URI, reading > > Object > > > > >>>> Storage, > > > > >>>> >> reading SQL, etc.) should each have its own utility package, > so > > > > they > > > > >>>> can be > > > > >>>> >> released independently without dependency problems popping up > > if > > > we > > > > >>>> need to > > > > >>>> >> break compatibility to one purpose. But more providers are > > > > >>>> exponentially > > > > >>>> >> more difficult to maintain, so I’d settle for one utility > > > provider > > > > >>>> for now > > > > >>>> >> and split further if needed in the future. > > > > >>>> >> > > > > >>>> >> Very much agree with this general statement. That's why I > > > generally > > > > >>>> do > > > > >>>> >> not like the "util" approach because common packaging > > introduces > > > > >>>> >> unnecessary coupling (you have to upgrade independent utils > > > > >>>> together). And > > > > >>>> >> when we have a common set of things that seem to make sense > to > > be > > > > >>>> released > > > > >>>> >> together when upgraded we should package them together in > > > > >>>> >> "common.<something concrete" (like we have with common.io > and > > > > >>>> common.sql). > > > > >>>> >> > > > > >>>> >> However - in this case, I think what Jens proposed (and I am > > > happy > > > > >>>> to try > > > > >>>> >> as well) is to attempt to use symbolic links - i.e. add the > > code > > > in > > > > >>>> >> `airflow.util` but then create a symbolic link in the > provider. > > > I > > > > >>>> tested > > > > >>>> >> it yesterday and it works as expected - i.e. such symbolic > link > > > is > > > > >>>> >> dereferenced and the provider package contains the python > file, > > > not > > > > >>>> >> symbolic link. That seems like a much more lightweight > approach > > > > that > > > > >>>> will > > > > >>>> >> serve the purpose of "common.util" much better. The only > thing > > we > > > > >>>> will have > > > > >>>> >> to take care of (and we can add it once the POC is > successful) > > is > > > > to > > > > >>>> add > > > > >>>> >> some pre-commit protection that those kind of symbolically > > linked > > > > >>>> util > > > > >>>> >> modules are imported in providers, from inside of those > > provider, > > > > >>>> not from > > > > >>>> >> airlfow, and make sure they are "standalone" (i.e. - as you > > > > >>>> mentioned - not > > > > >>>> >> depend on anything in airflow code). We could create a new > > > package > > > > >>>> for that > > > > >>>> >> in airlfow > > > > >>>> >> "airlfow.provider_utils" for example - and make sure (as next > > > step) > > > > >>>> that > > > > >>>> >> anything from that package is never directly imported by any > > > > >>>> provider, and > > > > >>>> >> whenever provider uses it, it should be symbolic link inside > > that > > > > >>>> provider. > > > > >>>> >> That's all automatable and we can prevent mistakes via > > > pre-commit. > > > > >>>> >> > > > > >>>> >> I think that might lead to a very lightweight approach where > we > > > > >>>> introduce > > > > >>>> >> new common functionality which is immediately reusable in > > > providers > > > > >>>> without > > > > >>>> >> the hassle of taking care about backwards compatibility, and > > > > >>>> managing the > > > > >>>> >> "common.util" provider. At the expense of a bit complex > > > pre-commit > > > > >>>> that > > > > >>>> >> will guard the usage of it. > > > > >>>> >> > > > > >>>> >> Seems that it might be the "Eat cake and have it too" way > that > > > > we've > > > > >>>> been > > > > >>>> >> looking for. > > > > >>>> >> > > > > >>>> >> J. > > > > >>>> >> > > > > >>>> >> On Thu, Feb 22, 2024 at 6:14 AM Tzu-ping Chung > > > > >>>> <t...@astronomer.io.invalid> > > > > >>>> >> wrote: > > > > >>>> >> > > > > >>>> >>> It would help in the sense mentioned in previous posts, yes. > > But > > > > one > > > > >>>> >> thing > > > > >>>> >>> I want to point out is, for the provider to actually be > > helpful, > > > > it > > > > >>>> must > > > > >>>> >> be > > > > >>>> >>> treated a bit differently from normal providers, but more > > like a > > > > >>>> separate > > > > >>>> >>> third-party dependency. Specifically, the provider should > not > > > > have a > > > > >>>> >>> dependency to Core Airflow, so it can be released and > depended > > > on > > > > >>>> more > > > > >>>> >>> flexibly. > > > > >>>> >>> > > > > >>>> >>> Ideally utilities for each purpose (parsing URI, reading > > Object > > > > >>>> Storage, > > > > >>>> >>> reading SQL, etc.) should each have its own utility package, > > so > > > > >>>> they can > > > > >>>> >> be > > > > >>>> >>> released independently without dependency problems popping > up > > if > > > > we > > > > >>>> need > > > > >>>> >> to > > > > >>>> >>> break compatibility to one purpose. But more providers are > > > > >>>> exponentially > > > > >>>> >>> more difficult to maintain, so I’d settle for one utility > > > provider > > > > >>>> for > > > > >>>> >> now > > > > >>>> >>> and split further if needed in the future. > > > > >>>> >>> > > > > >>>> >>> TP > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>>> On 22 Feb 2024, at 10:10, Scheffler Jens (XC-AS/EAE-ADA-T) > < > > > > >>>> >>> jens.scheff...@de.bosch.com.INVALID> wrote: > > > > >>>> >>>> > > > > >>>> >>>> @Uranusjr would this help as a pilot in your AIP-60 code to > > > parse > > > > >>>> and > > > > >>>> >>> validate URIs for datasets? > > > > >>>> >>>> > > > > >>>> >>>> Mit freundlichen Grüßen / Best regards > > > > >>>> >>>> > > > > >>>> >>>> Jens Scheffler > > > > >>>> >>>> > > > > >>>> >>>> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > > > > >>>> >>>> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 > > > > >>>> Stuttgart-Vaihingen | > > > > >>>> >>> GERMANY | www.bosch.com > > > > >>>> >>>> Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > > > >>>> >>> jens.scheff...@de.bosch.com > > > > >>>> >>>> > > > > >>>> >>>> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, > HRB > > > > 14000; > > > > >>>> >>>> Aufsichtsratsvorsitzender: Prof. Dr. Stefan > Asenkerschbaumer; > > > > >>>> >>>> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian > Fischer, > > > Dr. > > > > >>>> Markus > > > > >>>> >>> Forschner, > > > > >>>> >>>> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja > > > > Rückert > > > > >>>> >>>> > > > > >>>> >>>> -----Original Message----- > > > > >>>> >>>> From: Jarek Potiuk <ja...@potiuk.com> > > > > >>>> >>>> Sent: Donnerstag, 22. Februar 2024 00:53 > > > > >>>> >>>> To: dev@airflow.apache.org > > > > >>>> >>>> Subject: Re: [DISCUSS] Common.util provider? > > > > >>>> >>>> > > > > >>>> >>>> Yep. It could work with symbolic links. Tested it and with > > > flit - > > > > >>>> both > > > > >>>> >>> wheel and sdist packaged code such symbolically linked file > is > > > > >>>> >> dereferenced > > > > >>>> >>> and copy of the file is added there. It could be a nice way > of > > > > >>>> doing it. > > > > >>>> >>>> > > > > >>>> >>>> Maybe then worth trying next time if someone has a need? > > > > >>>> >>>> > > > > >>>> >>>> J > > > > >>>> >>>> > > > > >>>> >>>> On Thu, Feb 22, 2024 at 12:39 AM Scheffler Jens > > > > (XC-AS/EAE-ADA-T) < > > > > >>>> >>> jens.scheff...@de.bosch.com.invalid> wrote: > > > > >>>> >>>> > > > > >>>> >>>>>>>> As of additional dependency complexity between > providers > > > > >>>> actually > > > > >>>> >>>>>>>> the > > > > >>>> >>>>> additional dependency I think creates more problems than > the > > > > >>>> benefit… > > > > >>>> >>>>> would be cool if there would be an option to „inline“ > common > > > > code > > > > >>>> from > > > > >>>> >>>>> a single place but keep individual providers fully > > > independent… > > > > >>>> >>>>> > > > > >>>> >>>>>> Well, we already do a lot of inlining, so if we think we > > > > should > > > > >>>> do > > > > >>>> >>>>>> more, > > > > >>>> >>>>> we have mechanisms for that. We have pre-commits and > > release > > > > >>>> commands > > > > >>>> >>>>> that do a lot of that. Pre commits are inlining scripts in > > > > >>>> >>>>> Dockerfiles, shortening PyPI readme . The providers > > > __init__.py > > > > >>>> files > > > > >>>> >>>>> and changelogs and index documentation .rst (partially) > are > > > > >>>> generated > > > > >>>> >>>>> at release documentation preparation time, pyproject.toml > > for > > > > >>>> >>>>> providers are generated from common templates at package > > > > building > > > > >>>> time > > > > >>>> >>>>> and so on and so on :). So we can do more of that and > > generate > > > > >>>> common > > > > >>>> >>>>> code, it's just a matter of adding pre-commits or breeze > > > > scripts. > > > > >>>> But > > > > >>>> >>>>> again "can't have and eat cake" - this has the drawback > that > > > > >>>> there are > > > > >>>> >>>>> extra steps involved and even if it's automated it does > add > > > > >>>> friction > > > > >>>> >>>>> when you have to regenerate the code every time you change > > it > > > > and > > > > >>>> when > > > > >>>> >>>>> you change it in another place than where you use it. > > > > >>>> >>>>> > > > > >>>> >>>>> Yes, also thought a moment about pre-commit. I#d be okay > if > > we > > > > >>>> really > > > > >>>> >>>>> in-line and have a pre-commit aligning the redundancy of > > > python > > > > in > > > > >>>> >>> folders. > > > > >>>> >>>>> Might need to be an opt-in if only 10 of 85 providers are > > > using > > > > >>>> common > > > > >>>> >>>>> stuff and if we change a common line we probably do not > need > > > to > > > > >>>> affect > > > > >>>> >>>>> all providers. > > > > >>>> >>>>> > > > > >>>> >>>>> As long as no Windows users trying to code for airflow (do > > we > > > > >>>> need to > > > > >>>> >>>>> consider?) would it also work to use symlinks? Git can > cope > > > with > > > > >>>> this, > > > > >>>> >>>>> I don't know if the python toolchain can de-reference a > copy > > > and > > > > >>>> are > > > > >>>> >>>>> not packaging a symlink? Would be worth a test... would > save > > > the > > > > >>>> >>>>> pre-commit and we even could selectively include common > bla > > > into > > > > >>>> >>>>> providers as needed :-D > > > > >>>> >>>>> > > > > >>>> >>>>> Mit freundlichen Grüßen / Best regards > > > > >>>> >>>>> > > > > >>>> >>>>> Jens Scheffler > > > > >>>> >>>>> > > > > >>>> >>>>> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert > Bosch > > > > GmbH > > > > >>>> | > > > > >>>> >>>>> Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY > | > > > > >>>> >>>>> www.bosch.com Tel. +49 711 811-91508 | Mobil +49 160 > > > 90417410 | > > > > >>>> >>>>> jens.scheff...@de.bosch.com > > > > >>>> >>>>> > > > > >>>> >>>>> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, > HRB > > > > >>>> 14000; > > > > >>>> >>>>> Aufsichtsratsvorsitzender: Prof. Dr. Stefan > > Asenkerschbaumer; > > > > >>>> >>>>> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian > Fischer, > > > Dr. > > > > >>>> >>>>> Markus Forschner, Stefan Grosch, Dr. Markus Heyn, Dr. > Frank > > > > >>>> Meyer, Dr. > > > > >>>> >>>>> Tanja Rückert > > > > >>>> >>>>> > > > > >>>> >>>>> -----Original Message----- > > > > >>>> >>>>> From: Jarek Potiuk <ja...@potiuk.com> > > > > >>>> >>>>> Sent: Mittwoch, 21. Februar 2024 21:18 > > > > >>>> >>>>> To: dev@airflow.apache.org > > > > >>>> >>>>> Subject: Re: [DISCUSS] Common.util provider? > > > > >>>> >>>>> > > > > >>>> >>>>>> if we have a common piece then we are locking all > depending > > > > >>>> >>>>>> providers > > > > >>>> >>>>> (potentially) together if common code changes > > > > >>>> >>>>> > > > > >>>> >>>>> Yes, coupling in this case is the drawback of this > solution. > > > You > > > > >>>> can't > > > > >>>> >>>>> have cake and eat it too and in this case you trade DRY > with > > > > >>>> coupling. > > > > >>>> >>>>> > > > > >>>> >>>>>> As of additional dependency complexity between providers > > > > actually > > > > >>>> >>>>>> the > > > > >>>> >>>>> additional dependency I think creates more problems than > the > > > > >>>> benefit… > > > > >>>> >>>>> would be cool if there would be an option to „inline“ > common > > > > code > > > > >>>> from > > > > >>>> >>>>> a single place but keep individual providers fully > > > independent… > > > > >>>> >>>>> > > > > >>>> >>>>> Well, we already do a lot of inlining, so if we think we > > > should > > > > >>>> do > > > > >>>> >>>>> more, we have mechanisms for that. We have pre-commits > and > > > > >>>> release > > > > >>>> >>>>> commands that do a lot of that. Pre commits are inlining > > > scripts > > > > >>>> in > > > > >>>> >>>>> Dockerfiles, shortening PyPI readme . The providers > > > __init__.py > > > > >>>> files > > > > >>>> >>>>> and changelogs and index documentation .rst (partially) > are > > > > >>>> generated > > > > >>>> >>>>> at release documentation preparation time, pyproject.toml > > for > > > > >>>> >>>>> providers are generated from common templates at package > > > > building > > > > >>>> time > > > > >>>> >>>>> and so on and so on :). So we can do more of that and > > generate > > > > >>>> common > > > > >>>> >>>>> code, it's just a matter of adding pre-commits or breeze > > > > scripts. > > > > >>>> But > > > > >>>> >>>>> again "can't have and eat cake" - this has the drawback > that > > > > >>>> there are > > > > >>>> >>>>> extra steps involved and even if it's automated it does > add > > > > >>>> friction > > > > >>>> >>>>> when you have to regenerate the code every time you change > > it > > > > and > > > > >>>> when > > > > >>>> >>>>> you change it in another place than where you use it. > > > > >>>> >>>>> > > > > >>>> >>>>> J. > > > > >>>> >>>>> > > > > >>>> >>>>> On Wed, Feb 21, 2024 at 9:02 PM Scheffler Jens > > > > (XC-AS/EAE-ADA-T) < > > > > >>>> >>>>> jens.scheff...@de.bosch.com.invalid> wrote: > > > > >>>> >>>>> > > > > >>>> >>>>>> Hi Jarek, > > > > >>>> >>>>>> > > > > >>>> >>>>>> At reviewing the PR from uranusjr for AIP-60 I also had > the > > > > >>>> feeling > > > > >>>> >>>>>> that a lot of very similar code is repeated in all the > > > > providers. > > > > >>>> >>>>>> But during review yesterday I dropped the ides because if > > we > > > > >>>> have a > > > > >>>> >>>>>> common piece then we are locking all depending providers > > > > >>>> >>>>>> (potentially) together if common code changes. > > > > >>>> >>>>>> As of additional dependency complexity between providers > > > > actually > > > > >>>> >>>>>> the additional dependency I think creates more prblems > than > > > the > > > > >>>> >>>>>> benefit… would be cool if tehere would be an option to > > > „inline“ > > > > >>>> >>>>>> common code from a single place but keep individual > > providers > > > > >>>> fully > > > > >>>> >>>>>> independent… > > > > >>>> >>>>>> > > > > >>>> >>>>>> Jens > > > > >>>> >>>>>> > > > > >>>> >>>>>> Sent from Outlook for > > > > >>>> >>>>>> iOS< > > > > >>>> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F% > > > > >>>> >>>>>> 2F > > > > >>>> >>>>>> aka.ms%2Fo0ukef&data=05%7C02%7CJens.Scheffler% > > 40de.bosch.com > > > > >>>> %7C98c88 > > > > >>>> >>>>>> 97 > > > > >>>> >>>>>> > > > > >>>> > > 195d944d483ab08dc331a49bb%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0 > > > > >>>> >>>>>> %7 > > > > >>>> >>>>>> > > > > >>>> > > C638441435197193656%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ > > > > >>>> >>>>>> Ij > > > > >>>> >>>>>> > > > > >>>> > > oiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=n6gk9fNn > > > > >>>> >>>>>> WB SJOPYEgJ9WbriZ3H4id3RhLr16SguOuFA%3D&reserved=0> > > > > >>>> >>>>>> ________________________________ > > > > >>>> >>>>>> From: Jarek Potiuk <ja...@potiuk.com> > > > > >>>> >>>>>> Sent: Wednesday, February 21, 2024 5:42:20 PM > > > > >>>> >>>>>> To: dev@airflow.apache.org <dev@airflow.apache.org> > > > > >>>> >>>>>> Subject: [DISCUSS] Common.util provider? > > > > >>>> >>>>>> > > > > >>>> >>>>>> Hello everyone, > > > > >>>> >>>>>> > > > > >>>> >>>>>> How do we feel about introducing a common.util provider? > > > > >>>> >>>>>> > > > > >>>> >>>>>> I know it's not been the original idea behind providers, > > but > > > - > > > > >>>> after > > > > >>>> >>>>>> testing common.sql and now also having common.io, seems > > like > > > > >>>> more > > > > >>>> >>>>>> and more we would like to extract some common code that > we > > > > would > > > > >>>> >>>>>> like providers to use, but we refrain from it, because it > > > will > > > > >>>> only > > > > >>>> >>>>>> be actually usable 6 months after we introduce some > common > > > > code. > > > > >>>> >>>>>> > > > > >>>> >>>>>> However, if we introduce common.util, this problem is > > > generally > > > > >>>> gone > > > > >>>> >>>>>> - at the expense of more complex maintenance and > > > cross-provider > > > > >>>> >>>>> dependencies. > > > > >>>> >>>>>> We should be able to add a common util method and use it > > in a > > > > >>>> >>>>>> provider at the same time. > > > > >>>> >>>>>> > > > > >>>> >>>>>> Think Amazon provider using a new feature released in > > > > common.util > > > > >>>> >>>>>>> =1.2.0 and google provider >= 1.1.0. All manageable and > we > > > do > > > > it > > > > >>>> >>>>>> already for common.sql. We know how to do it, we know > what > > to > > > > >>>> avoid, > > > > >>>> >>>>>> we know we cannot introduce backwards-incompatible > changes, > > > so > > > > we > > > > >>>> >>>>>> have to be very clear what is and what is not a public > API > > > > >>>> there, We > > > > >>>> >>>>>> could rather easily add tests to prevent such > > > > >>>> backwards-incompatible > > > > >>>> >>>>>> changes. We even have a solution for chicken-egg > providers > > > > where > > > > >>>> we > > > > >>>> >>>>>> need to release two providers at the same time if they > > depend > > > > on > > > > >>>> >>>>>> each other. Generally speaking it's quite workable but > > adds a > > > > >>>> bit of > > > > >>>> >>> overhead. > > > > >>>> >>>>>> > > > > >>>> >>>>>> Examples that we could implement as "common.util": > > > > >>>> >>>>>> > > > > >>>> >>>>>> - common versioning check with cache - where multiple > > > providers > > > > >>>> >>>>>> could reuse "do we have pendulum 2" > > > > >>>> >>>>>> - more complex - some date management features (we have a > > few > > > > >>>> like > > > > >>>> >>>>>> date_ranges/round_time). But there are many more. > > > > >>>> >>>>>> > > > > >>>> >>>>>> I generally do not love the common "util" approach. It > has > > a > > > > >>>> >>>>>> tendency to become a bag of everything over time. but if > we > > > > >>>> limit it > > > > >>>> >>>>>> to a set of small, fully decoupled modules where each > > module > > > is > > > > >>>> >>>>>> independent - it's OK. And we already have it in > > > "airflow.util" > > > > >>>> and > > > > >>>> >>>>>> we seem to be > > > > >>>> >>>>> doing well. > > > > >>>> >>>>>> > > > > >>>> >>>>>> WDYT? Is it worth it ? > > > > >>>> >>>>>> > > > > >>>> >>>>>> J. > > > > >>>> >>>>>> > > > > >>>> >>>>> > > > > >>>> >>>> > > > > >>>> >>>> > > > > >>>> > > > --------------------------------------------------------------------- > > > > >>>> >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > >>>> >>>> For additional commands, e-mail: > dev-h...@airflow.apache.org > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >> > > > > >>>> > > > > >>>> > > > > >>>> > > > --------------------------------------------------------------------- > > > > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >>>> > > > > >>>> > > > > > > > > > >
