Hello everyone, This is a call for the vote to make an internal change to move the code of K8S, Celery and related (LocalKubernetes., CeleryKubernetes etc. ) to respective providers.
Consider it +1 (binding) from my side. This has been discussed in https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do and let me summarize it below: # Why? Multiple reasons: * It will make it easier to manage consistency between K8S Pod Manager and K8S executor. In the past there were non-trivial dependencies between those that resulted in k8s provider being limited to latest airflow versions * It's non-obvious that the code used in K8S executor uses two different artifacts (airflow and cncnf.k8s provider) and it limits our abilities to refactor/modify/improve this code as it has to work with various combinations of airflow + cncf.kubernete versions * provider's releases (major/minor versions) have much faster release cycle and we can both - fix and provide new features to those executors * users who have good reasons to not to upgrade to latest airflow releases will be able to use latest k8s/celery executors by updating providers only * if there are regressions with executors in newer airflow versions, users will be able to downgrade providers - without downgrading the whole airflow (downgrading the DB etc.) * this follows the philosophy of Airflow-as-a-platform, where anyone can extend Airflow by adding new plugins/providers and moving the executor to providers proves the point that anyone can do their own executor and that they will have the same capabilities as the ones that are built-in # Why now? We are in the process of finishing AIP-51 with executor decoupling and where we got rid of the hard-coded behaviour of Airflow depending on what executor was used. It was simply impossible before to move the executors to providers, because the hard-coded behaviours had to maintain the knowledge about which executor is used. Executor's API was incomplete and some behaviours of the executors were hard-coded. With AIP-51 completed executor implementation can simply rely on the complete executor's API - including exposing properties of the executor that can change airflow core behaviour appropriately by inspecting the properties. # Backwards compatibility I believe we will be able to make it fully backwards compatible with the usage of PEP 562 and deprecation notices (same as we did with contrib packages). Also we seem to be converging on the backwards-compatibility approach, specifically excluding the implementation of executors from our "Public API list" https://lists.apache.org/thread/d90b1yvsbwzy5flnd3vslfjs38x76kyj We will turn "cncf.kubernetes" and "celery" providers into "pre-installed" providers, which means that one will be able to use all the built-in executors with just "pip install airflow" (interestingly enough before that one had to install the k8s provider to make the K8s executor work even if they were part of the core which was sub-optimal). Also, resulting from the discussion we will keep documentation for available executors in Airflow (so they will still be considered as THE executors available and will be discoverable in the same way as today). # Potential problems Seems there are no known problems it can cause. There is the question "where to put CeleryKubernetesExecutor?" and the proposal is to put it in "cncf.k8s" and treat celery as an optional dependency ("celery" extra) of "cncf.k8s" provider. Since both providers will be pre-installed, this is not a problem or concern for any use case. J.
