as there were no concerns raised I started a lazy consensus https://lists.apache.org/thread/xmqzqzk90v0wyj5crm191wsjh8y1q2r2
On Mon, Sep 11, 2023 at 4:15 PM Oliveira, Niko <oniko...@amazon.com.invalid> wrote: > +1 to this! > > I also have a docs section half written on the executor interface and how > to extend it. But I've been very busy with a few other items that are > completing soon. > > Cheers, > Niko > > ________________________________ > From: Pankaj Koti <pankaj.k...@astronomer.io.INVALID> > Sent: Friday, September 8, 2023 12:03:35 PM > To: dev@airflow.apache.org > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Executors docs should > be published in Airflow core or providers? > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > +1 to the proposal. > > I think core Airflow docs can contain details about the default executor > that > gets shipped with standalone Airflow installation and a short note about > possibilities of using other (providers) executors in production and saying > to look > for detailed docs in the corresponding provider. > > Regards, > > > > Pankaj Koti > > *Senior Software Engineer, *OSS Engineering Team. > Location: Pune, India > > Timezone: Indian Standard Time (IST) > > Email: pankaj.k...@astronomer.io > > Mobile: +91 9730079985 > > > On Fri, Sep 8, 2023 at 11:28 PM Hussein Awala <huss...@awala.fr> wrote: > > > Since we moved the executors to the providers packages and made the > > executor interface pluggable and extensible, we should move the docs to > > their corresponding providers. However, we need to keep a doc in Airflow > > core that explains how to use/configure a provider executor (as we have > for > > the secret managers and the task log handlers) and maybe how to create a > > new custom one. > > > > On Fri, Sep 8, 2023 at 6:15 PM Elad Kalif <elad...@apache.org> wrote: > > > > > Hello everyone, > > > > > > This thread is opened due to open issue Migrate Celery/Dask/Kubernetes > > > Executor docs to providers < > > https://github.com/apache/airflow/issues/33916 > > > > > > > > > > *Background:* > > > We had a discussion about extracting Celery, Kubernetes, Dask executors > > > from core to providers (discussion thread > > > <https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>, > vote > > > thread < > https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8 > > >) > > > > > > One of the things we voted on was: > > > > > > Also, resulting from the discussion we will keep documentation for > > > > available executors in Airflow (so they will still be considered as > THE > > > > executors available and will be discoverable in the same way as > today). > > > > > > > > > *The problem:* > > > Airflow Core and Providers do not share the same release cycle nor > > cadance. > > > This means that if we add new capabilities to executors or fix an issue > > > which requires both code and doc update - the code will be delivered > > > before/ahead of documentation. Both cases are not good. > > > > > > > > > *My proposal:*Now that Celery, Kubernetes, Dask executors are in > > providers. > > > The section of core-concept > > > < > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html > > > > > > > should > > > contain only HIGH level information about the executors. It should not > > > contain information about executors internals or how to address common > > > problems. This information should be in the provider docs. The high > level > > > info should be short and on a level that is relevant for all executors > > > which means it's not likely to have many changes over time. The > > > core-concept should have links/refers for deep dive read to the > provider > > > docs. This is very similar to what we do with Notifiers > > > < > > > > > > https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html > > > > > > > core > > > contains high level information and a list of notifers that are linked > to > > > provider docs. > > > > > > WDYT? > > > > > > Also, resulting from the discussion we will keep documentation for > > > available executors in Airflow (so they will still be considered as THE > > > executors available and will be discoverable in the same way as > > today).Moe > > > K8S and Celery Executors (and related) to respective providers? > > > > > >
