I'll have to read your proposal in detail (sorry, no time right now!), but I'm broadly in favour of this approach, and I think keeping them _in_ the same repo is the best plan -- that makes writing and testing cross-cutting changes easier.
-a > On 28 Oct 2019, at 12:14, Tomasz Urbaszek <tomasz.urbas...@polidea.com> wrote: > > I think utilizing namespaces should reduce a lot of problems raised by > using separate repos (who will manage it? how to release? where should be > the repo?). > > Bests, > Tomek > > On Sun, Oct 27, 2019 at 11:54 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >> Thanks Bas for comments! Let me share my thoughts below. >> >> On Sun, Oct 27, 2019 at 9:23 AM Bas Harenslak < >> basharens...@godatadriven.com> >> wrote: >> >>> Hi Jarek, I definitely see a future in creating separate installable >>> packages for various operators/hooks/etc (as in AIP-8). This would IMO >>> strip the “core” Airflow to only what’s needed and result in a small >>> package without a ton of dependencies (and make it more maintainable, >>> shorter tests, etc etc etc). Not exactly sure though what you’re >> proposing >>> in your e-mail, is it a new AIP for an intermediate step towards AIP-8? >>> >> >> It's a new AIP I am proposing. For now it's only for backporting the new >> 2.0 import paths to 1.10.* series. >> >> It's more of "incremental going in direction of AIP-8 and learning some >> difficulties involved" than implementing AIP-8 fully. We are taking >> advantage of changes in import paths from AIP-21 which make it possible to >> have both old and new (optional) operators available in 1.10.* series of >> Airflow. I think there is a lot more to do for full implementation of >> AIP-8: decisions how to maintain, install those operator groups separately, >> stewardship model/organisation for the separate groups, how to manage >> cross-dependencies, procedures for releasing the packages etc. >> >> I think about this new AIP also as a learning effort - we would learn more >> how separate packaging works and then we can follow up with AIP-8 full >> implementation for "modular" Airflow. Then AIP-8 could be implemented in >> Airflow 2.1 for example - or 3.0 if we start following semantic versioning >> - based on those learnings. It's a bit of good example of having cake and >> eating it too. We can try out modularity in 1.10.* while cutting the scope >> of 2.0 and not implementing full management/release procedure for AIP-8 >> yet. >> >> >>> Thinking about this, I think there are still a few grey areas (which >> would >>> be good to discuss in a new AIP, or continue on AIP-8): >>> >>> * In your email you only speak only about the 3 big cloud providers >>> (btw I made a PR for migrating all AWS components -> >>> https://github.com/apache/airflow/pull/6439). Is there a plan for >>> splitting other components than Google/AWS/Azure? >>> >> >> We could add more groups as part of this new AIP indeed (as an extension to >> AIP-21 and pre-requisite to AIP-8). We already see how moving/deprecation >> works for the providers package - it works for GCP/Google rather nicely. >> But there is nothing to prevent us from extending it to cover other groups >> of operators/hooks. If you look at the current structure of documentation >> done by Kamil, we can follow the structure there and move the >> operators/hooks accordingly ( >> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html): >> >> Fundamentals, ASF: Apache Software Foundation, Azure: Microsoft >> Azure, AWS: Amazon Web Services, GCP: Google Cloud Platform, Service >> integrations, Software integrations, Protocol integrations. >> >> I am happy to include that in the AIP - if others agree it's a good idea. >> Out of those groups - I think only Fundamentals should not be back-ported. >> Others should be rather easy to port (if we decide to). We already have >> quite a lot of those in the new GCP operators for 2.0. So starting with >> GCP/Google group is a good idea. Also following with Cloud Providers first >> is a good thing. For example we have now support from Google Composer team >> to do this separation for GCP (and we learn from it) and then we can claim >> the stewardship in our team for releasing the python 3/ Airflow >> 1.10-compatible "airflow-google" packages. Possibly other Cloud >> Providers/teams might follow this (if they see the value in it) and there >> could be different stewards for those. And then we can do other groups if >> we decide to. I think this way we can learn whether AIP-8 is manageable and >> what real problems we are going to face. >> >> * Each “plugin” e.g. GCP would be a separate repo, should we create >>> some sort of blueprint for such packages? >>> >> >> I think we do not need separate repos (at all) but in this new AIP we can >> test it before we decide to go for AIP-8. IMHO - monorepo approach will >> work here rather nicely. We could use python-3 native namespaces >> <https://packaging.python.org/guides/packaging-namespace-packages/> for >> the >> sub-packages when we go full AIP-8. For now we could simply package the new >> operators in separate pip package for Python 3 version 1.10.* series only. >> We only need to test if it works well with another package providing >> 'airflow.providers.*' after apache-airflow is installed (providing >> 'airflow' package). But I think we can make it work. I don't think we >> really need to split the repos, namespaces will work just fine and has >> easier management of cross-repository dependencies (but we can learn >> otherwise). For sure we will not need it for the new proposed AIP of >> backporting groups to 1.10 and we can defer that decision to AIP-8 >> implementation time. >> >> >>> * In which Airflow version do we start raising deprecation warnings >>> and in which version would we remove the original? >>> >> >> I think we should do what we did in GCP case already. Those old "imports" >> for operators can be made as deprecated in Airflow 2.0 (and removed in 2.1 >> or 3.0 if we start following semantic versioning). We can however do it >> before in 1.10.7 or 1.10.8 if we release those (without removing the old >> operators yet - just raise deprecation warnings and inform that for python3 >> the new "airflow-google", "airflow-aws" etc. packages can be installed and >> users can switch to it). >> >> J. >> >> >>> >>> Cheers, >>> Bas >>> >>> On 27 Oct 2019, at 08:33, Jarek Potiuk <jarek.pot...@polidea.com<mailto: >>> jarek.pot...@polidea.com>> wrote: >>> >>> Hello - any comments on that? I am happy to make it into an AIP :)? >>> >>> On Sun, Oct 13, 2019 at 5:53 PM Jarek Potiuk <jarek.pot...@polidea.com >>> <mailto:jarek.pot...@polidea.com>> >>> wrote: >>> >>> *Motivation* >>> >>> I think we really should start thinking about making it easier to migrate >>> to 2.0 for our users. After implementing some recent changes related to >>> AIP-21- >>> Changes in import paths >>> < >>> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths >>> >>> I >>> think I have an idea that might help with it. >>> >>> *Proposal* >>> >>> We could package some of the new and improved 2.0 operators (moved to >>> "providers" package) and let them be used in Python 3 environment of >>> airflow 1.10.x. >>> >>> This can be done case-by-case per "cloud provider". It should not be >>> obligatory, should be largely driven by each provider. It's not yet full >>> AIP-8 >>> Split Hooks/Operators into separate packages >>> < >>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303 >>> . >>> It's >>> merely backporting of some operators/hooks to get it work in 1.10. But by >>> doing it we might try out the concept of splitting, learn about >> maintenance >>> problems and maybe implement full *AIP-8 *approach in 2.1 consistently >>> across the board. >>> >>> *Context* >>> >>> Part of the AIP-21 was to move import paths for Cloud providers to >>> separate providers/<PROVIDER> package. An example for that (the first >>> provider we already almost migrated) was providers/google package >> (further >>> divided into gcp/gsuite etc). >>> >>> We've done a massive migration of all the Google-related operators, >>> created a few missing ones and retrofitted some old operators to follow >> GCP >>> best practices and fixing a number of problems - also implementing >> Python3 >>> and Pylint compatibility. Some of these operators/hooks are not backwards >>> compatible. Those that are compatible are still available via the old >>> imports with deprecation warning. >>> >>> We've added missing tests (including system tests) and missing features - >>> improving some of the Google operators - giving the users more >> capabilities >>> and fixing some issues. Those operators should pretty much "just work" in >>> Airflow 1.10.x (any recent version) for Python 3. We should be able to >>> release a separate pip-installable package for those operators that users >>> should be able to install in Airflow 1.10.x. >>> >>> Any user will be able to install this separate package in their Airflow >>> 1.10.x installation and start using those new "provider" operators in >>> parallel to the old 1.10.x operators. Other providers ("microsoft", >>> "amazon") might follow the same approach if they want. We could even at >>> some point decide to move some of the core operators in similar fashion >>> (for example following the structure proposed in the latest >> documentation: >>> fundamentals / software / etc. >>> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html) >>> >>> *Pros and cons* >>> >>> There are a number of pros: >>> >>> - Users will have an easier migration path if they are deeply vested >>> into 1.10.* version >>> - It's possible to migrate in stages for people who are also vested in >>> py2: *py2 (1.10) -> py3 (1.10) -> py3 + new operators (1.10) -> py3 + >>> 2.0* >>> - Moving to new operators in py3 + new operators can be done >>> gradually. Old operators will continue to work while new can be used >> more >>> and more >>> - People will get incentivised to migrate to python 3 before 2.0 is >>> out (by using new operators) >>> - Each provider "package" can have independent release schedule - and >>> add functionality in already released Airflow versions. >>> - We do not take out any functionality from the users - we just add >>> more options >>> - The releases can be - similarly as main airflow releases - voted >>> separately by PMC after "stewards" of the package (per provider) >> perform >>> round of testing on 1.10.* versions. >>> - Users will start migrating to new operators earlier and have >>> smoother switch to 2.0 later >>> - The latest improved operators will start >>> >>> There are three cons I could think of: >>> >>> - There will be quite a lot of duplication between old and new >>> operators (they will co-exist in 1.10). That might lead to confusion of >>> users and problems with cooperation between different operators/hooks >>> - Having new operators in 1.10 python 3 might keep people from >>> migrating to 2.0 >>> - It will require some maintenance and separate release overhead. >>> >>> I already spoke to Composer team @Google and they are very positive about >>> this. I also spoke to Ash and seems it might also be OK for Astronomer >>> team. We have Google's backing and support, and we can provide >> maintenance >>> and support for those packages - being an example for other providers how >>> they can do it. >>> >>> Let me know what you think - and whether I should make it into an >> official >>> AIP maybe? >>> >>> J. >>> >>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> > > > -- > > Tomasz Urbaszek > Polidea <https://www.polidea.com/> | Junior Software Engineer > > M: +48 505 628 493 <+48505628493> > E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work>
