+ 1 non-binding From: Elad Kalif <elad...@apache.org> Reply-To: "dev@airflow.apache.org" <dev@airflow.apache.org> Date: Tuesday, February 21, 2023 at 10:44 AM To: "dev@airflow.apache.org" <dev@airflow.apache.org> Subject: RE: [EXTERNAL][VOTE] Move K8S / Celery (and related) executors to respective providers
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. +1 (binding) On Tue, Feb 21, 2023 at 8:33 PM Josh Fell <josh.d.f...@astronomer.io.invalid> wrote: +1 binding On Tue, Feb 21, 2023 at 11:21 AM Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: And just to add as I missed that in the original mail - this is "code modification" vote - so all committers have a binding vote. https://www.apache.org/foundation/voting.html#votes-on-code-modification Also I have not mentioned the time: I think we can keep it open for 72 hours from now - which means that it will end on February 24th, 2023, 5PM CET. On Tue, Feb 21, 2023 at 4:26 PM Ash Berlin-Taylor <a...@apache.org<mailto:a...@apache.org>> wrote: > > +1 binding > > On Feb 21 2023, at 9:21 am, Hussein Awala > <huss...@awala.fr<mailto:huss...@awala.fr>> wrote: > > > +1 non-binding.I'm a little concerned that this coupling will reduce the fast > evolution of providers, but given the benefits on the executor side, I vote > for it. > From: Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> > Sent: Tuesday, February 21, 2023 8:45:15 AM > To: dev@airflow.apache.org<mailto:dev@airflow.apache.org> > Subject: [VOTE] Move K8S / Celery (and related) executors to respective > providers > > Hello everyone, > > This is a call for the vote to make an internal change to move the code of > K8S, Celery and related (LocalKubernetes., CeleryKubernetes etc. ) to > respective providers. > > Consider it +1 (binding) from my side. > > This has been discussed in > https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do and let me > summarize it below: > > # Why? > > Multiple reasons: > > * It will make it easier to manage consistency between K8S Pod Manager and > K8S executor. In the past there were non-trivial dependencies between those > that resulted in k8s provider being limited to latest airflow versions > * It's non-obvious that the code used in K8S executor uses two different > artifacts (airflow and cncnf.k8s provider) and it limits our abilities to > refactor/modify/improve this code as it has to work with various combinations > of airflow + cncf.kubernete versions > * provider's releases (major/minor versions) have much faster release cycle > and we can both - fix and provide new features to those executors > * users who have good reasons to not to upgrade to latest airflow releases > will be able to use latest k8s/celery executors by updating providers only > * if there are regressions with executors in newer airflow versions, users > will be able to downgrade providers - without downgrading the whole airflow > (downgrading the DB etc.) > * this follows the philosophy of Airflow-as-a-platform, where anyone can > extend Airflow by adding new plugins/providers and moving the executor to > providers proves the point that anyone can do their own executor and that > they will have the same capabilities as the ones that are built-in > > # Why now? > > We are in the process of finishing AIP-51 with executor decoupling and where > we got rid of the hard-coded behaviour of Airflow depending on what executor > was used. It was simply impossible before to move the executors to providers, > because the hard-coded behaviours had to maintain the knowledge about which > executor is used. Executor's API was incomplete and some behaviours of the > executors were hard-coded. With AIP-51 completed executor implementation can > simply rely on the complete executor's API - including exposing properties of > the executor that can change airflow core behaviour appropriately by > inspecting the properties. > > # Backwards compatibility > > I believe we will be able to make it fully backwards compatible with the > usage of PEP 562 and deprecation notices (same as we did with contrib > packages). Also we seem to be converging on the backwards-compatibility > approach, specifically excluding the implementation of executors from our > "Public API list" > https://lists.apache.org/thread/d90b1yvsbwzy5flnd3vslfjs38x76kyj > > We will turn "cncf.kubernetes" and "celery" providers into "pre-installed" > providers, which means that one will be able to use all the built-in > executors with just "pip install airflow" (interestingly enough before that > one had to install the k8s provider to make the K8s executor work even if > they were part of the core which was sub-optimal). > > Also, resulting from the discussion we will keep documentation for available > executors in Airflow (so they will still be considered as THE executors > available and will be discoverable in the same way as today). > > # Potential problems > > Seems there are no known problems it can cause. There is the question "where > to put CeleryKubernetesExecutor?" and the proposal is to put it in "cncf.k8s" > and treat celery as an optional dependency ("celery" extra) of "cncf.k8s" > provider. Since both providers will be pre-installed, this is not a problem > or concern for any use case. > > J.
