+1 to the proposal. I think core Airflow docs can contain details about the default executor that gets shipped with standalone Airflow installation and a short note about possibilities of using other (providers) executors in production and saying to look for detailed docs in the corresponding provider.
Regards, Pankaj Koti *Senior Software Engineer, *OSS Engineering Team. Location: Pune, India Timezone: Indian Standard Time (IST) Email: pankaj.k...@astronomer.io Mobile: +91 9730079985 On Fri, Sep 8, 2023 at 11:28 PM Hussein Awala <huss...@awala.fr> wrote: > Since we moved the executors to the providers packages and made the > executor interface pluggable and extensible, we should move the docs to > their corresponding providers. However, we need to keep a doc in Airflow > core that explains how to use/configure a provider executor (as we have for > the secret managers and the task log handlers) and maybe how to create a > new custom one. > > On Fri, Sep 8, 2023 at 6:15 PM Elad Kalif <elad...@apache.org> wrote: > > > Hello everyone, > > > > This thread is opened due to open issue Migrate Celery/Dask/Kubernetes > > Executor docs to providers < > https://github.com/apache/airflow/issues/33916 > > > > > > > *Background:* > > We had a discussion about extracting Celery, Kubernetes, Dask executors > > from core to providers (discussion thread > > <https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>, vote > > thread <https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8 > >) > > > > One of the things we voted on was: > > > > Also, resulting from the discussion we will keep documentation for > > > available executors in Airflow (so they will still be considered as THE > > > executors available and will be discoverable in the same way as today). > > > > > > *The problem:* > > Airflow Core and Providers do not share the same release cycle nor > cadance. > > This means that if we add new capabilities to executors or fix an issue > > which requires both code and doc update - the code will be delivered > > before/ahead of documentation. Both cases are not good. > > > > > > *My proposal:*Now that Celery, Kubernetes, Dask executors are in > providers. > > The section of core-concept > > < > > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html > > > > > should > > contain only HIGH level information about the executors. It should not > > contain information about executors internals or how to address common > > problems. This information should be in the provider docs. The high level > > info should be short and on a level that is relevant for all executors > > which means it's not likely to have many changes over time. The > > core-concept should have links/refers for deep dive read to the provider > > docs. This is very similar to what we do with Notifiers > > < > > > https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html > > > > > core > > contains high level information and a list of notifers that are linked to > > provider docs. > > > > WDYT? > > > > Also, resulting from the discussion we will keep documentation for > > available executors in Airflow (so they will still be considered as THE > > executors available and will be discoverable in the same way as > today).Moe > > K8S and Celery Executors (and related) to respective providers? > > >
