Clarification: Ash is right (I totally forgot about it). We do have dask in https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html#core-airflow-extras and it even explicitly mentions it:
* "dask pip install 'apache-airflow[dask]' DaskExecutor" So technically speaking you **should** use the dask extra to install distributed, dask and cloudpickle package - all needed to run the dask executor. And we will keep it this way - it will just move from "core extras" to "providers". Actually it will move to "deprecated" and we will alias "dask" it with the new "daskexecutor", because the name "apache-airflow-providers-dask" is not allowed by PyPI (I guess the -dask suffix is considered potentially harmful for typosquatting) so I reserved "apache-airflow-providers-daskexecutor". J. On Fri, Jul 21, 2023 at 10:34 AM Hussein Awala <huss...@awala.fr> wrote: > > > based on the premise that your had to install the `dask` extra in the > first place to get dask module of the right version > > After reviewing the Dask Executor > <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html> > section in Airflow documentation, I didn't find any requirement or > recommendation to add the `dask` extra when using it. So, including the > dependencies explicitly in the requirements file should suffice to use the > executor. This means that this change might potentially break existing > setups for some users when they upgrade Airflow version, which contradicts > the Airflow deprecation policy. > > If the premise holds true for Dask, it should also apply to Celery and > Kubernetes. Nevertheless, we decided to pre-install their providers after > the migration. > > However, this executor isn't as popular as the others, and as an Airflow > user, I prefer not to have Dask dependencies in my environment when I > install Airflow with the Celery executor. This helps avoid conflicts and > enables me to upgrade libraries independently from their constraints. One > possible solution could be allowing users to instruct Airflow not to > pre-install executors providers by passing an argument via > `--install-option` during pip install, and add install them explicitly by > providing the extras. > > Since the flexibility of our deprecation policy isn't entirely clear to me, > I'm neutral on this matter, with a vote of +0. > > On Fri, Jul 21, 2023 at 9:10 AM Elad Kalif <elad...@apache.org> wrote: > > > I agree with Ash > > -1 as well. > > > > > > On Fri, Jul 21, 2023 at 9:29 AM Ash Berlin-Taylor <a...@apache.org> wrote: > > > > > -1 - based on the premise that your had to install the `dask` extra in > > the > > > first place to get dask module of the right version, so if we make the > > > existing extra depends on the new provider then it's good enough. > > > > > > On 21 July 2023 06:22:28 BST, Jarek Potiuk <ja...@potiuk.com> wrote: > > > >Q: Do we want to pre-install Dask provider in Airflow 2.7.0 (with Dask > > > >executor) once separated ? > > > > > > > >Discussion was here: > > > >https://lists.apache.org/thread/0d9x4kl7hc2qzvho2mbdf35ohn5w12l6 > > > > > > > >Please vote: > > > > > > > >* +1 -> yes, we want to have dask provider preinstalled > > > >* -1 -> no, it's fine to make it optional > > > >* 0 -> no opinion > > > > > > > >Consider it my -1: I think we should NOT preinstall Dask provider. > > > > > > > >Voting guidelines here. This is really a "procedural" matter rather > > > >than code modification: > > > > > > > >> Votes on procedural issues follow the common format of majority rule > > > unless otherwise stated. That is, if there are more favourable votes than > > > unfavourable ones, the issue is considered to have passed -- regardless > > of > > > the number of votes in each category. (If the number of votes seems too > > > small to be representative of a community consensus, the issue is > > typically > > > not pursued. > > > > > > > >In this case committers have binding votes but other community members > > > >are encouraged to state their non-binding votes as well. > > > > > > > >https://www.apache.org/foundation/voting.html > > > > > > > >--------------------------------------------------------------------- > > > >To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
