+1 (binding) On Wed 22 Feb 2023 at 15:37, Kaxil Naik <kaxiln...@gmail.com> wrote:
> +1 binding > > On Wed, 22 Feb 2023 at 07:41, Ping Zhang <pin...@umich.edu> wrote: > >> +1 binding >> >> On Mon, Feb 20, 2023 at 23:46 Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> Hello everyone, >>> >>> This is a call for the vote to make an internal change to move the code >>> of K8S, Celery and related (LocalKubernetes., CeleryKubernetes etc. ) to >>> respective providers. >>> >>> Consider it +1 (binding) from my side. >>> >>> This has been discussed in >>> https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do and >>> let me summarize it below: >>> >>> # Why? >>> >>> Multiple reasons: >>> >>> * It will make it easier to manage consistency between K8S Pod Manager >>> and K8S executor. In the past there were non-trivial dependencies between >>> those that resulted in k8s provider being limited to latest airflow versions >>> * It's non-obvious that the code used in K8S executor uses two different >>> artifacts (airflow and cncnf.k8s provider) and it limits our abilities to >>> refactor/modify/improve this code as it has to work with various >>> combinations of airflow + cncf.kubernete versions >>> * provider's releases (major/minor versions) have much faster release >>> cycle and we can both - fix and provide new features to those executors >>> * users who have good reasons to not to upgrade to latest airflow >>> releases will be able to use latest k8s/celery executors by updating >>> providers only >>> * if there are regressions with executors in newer airflow versions, >>> users will be able to downgrade providers - without downgrading the whole >>> airflow (downgrading the DB etc.) >>> * this follows the philosophy of Airflow-as-a-platform, where anyone can >>> extend Airflow by adding new plugins/providers and moving the executor to >>> providers proves the point that anyone can do their own executor and that >>> they will have the same capabilities as the ones that are built-in >>> >>> # Why now? >>> >>> We are in the process of finishing AIP-51 with executor decoupling and >>> where we got rid of the hard-coded behaviour of Airflow depending on what >>> executor was used. It was simply impossible before to move the executors to >>> providers, because the hard-coded behaviours had to maintain the knowledge >>> about which executor is used. Executor's API was incomplete and some >>> behaviours of the executors were hard-coded. With AIP-51 completed executor >>> implementation can simply rely on the complete executor's API - including >>> exposing properties of the executor that can change airflow core behaviour >>> appropriately by inspecting the properties. >>> >>> # Backwards compatibility >>> >>> I believe we will be able to make it fully backwards compatible with the >>> usage of PEP 562 and deprecation notices (same as we did with contrib >>> packages). Also we seem to be converging on the >>> backwards-compatibility approach, specifically excluding the implementation >>> of executors from our "Public API list" >>> https://lists.apache.org/thread/d90b1yvsbwzy5flnd3vslfjs38x76kyj >>> >>> We will turn "cncf.kubernetes" and "celery" providers into >>> "pre-installed" providers, which means that one will be able to use all the >>> built-in executors with just "pip install airflow" (interestingly enough >>> before that one had to install the k8s provider to make the K8s executor >>> work even if they were part of the core which was sub-optimal). >>> >>> Also, resulting from the discussion we will keep documentation for >>> available executors in Airflow (so they will still be considered as THE >>> executors available and will be discoverable in the same way as today). >>> >>> # Potential problems >>> >>> Seems there are no known problems it can cause. There is the question >>> "where to put CeleryKubernetesExecutor?" and the proposal is to put it in >>> "cncf.k8s" and treat celery as an optional dependency ("celery" extra) of >>> "cncf.k8s" provider. Since both providers will be pre-installed, this is >>> not a problem or concern for any use case. >>> >>> J. >>> >>> -- >> >> Thanks, >> >> Ping >> >
