0 (non-binding) I agree with most of what has been discussed in the thread https://lists.apache.org/thread/0d9x4kl7hc2qzvho2mbdf35ohn5w12l6 and this email thread.
I did a check on Airflow discussions ( https://github.com/apache/airflow/discussions?discussions_q=dask+) and Airflow issues( https://github.com/apache/airflow/issues?q=is%3Aissue+dask+) and upon reading their descriptions, see that there are some users using the Dask executor although I personally have never tried it. If we decide to not pre-install it, in my opinion, a detailed and top-level news fragment / Changelog would help us here which mentions it to install the provider additionally in case they would like to continue using it. Regards, Pankaj Koti *Senior Software Engineer, *OSS Engineering Team. Location: Pune, India Timezone: Indian Standard Time (IST) Email: pankaj.k...@astronomer.io Mobile: +91 9730079985 On Fri, Jul 21, 2023 at 2:20 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > One possible solution could be allowing users to instruct Airflow not to > pre-install executors providers by passing an argument via > `--install-option` during pip install, and add install them explicitly by > providing the extras. > > Yes it would be nice. Unfortunately the dependency mechanisms of > Python (and `pip`) are not as flexible. You cannot individually remove > requirements, and having requirements is the only way you can have > "install also that other package" feature (extras is the mechanism to > install optional dependencies, but it's not something we should ask > regular users to do > > > > On Fri, Jul 21, 2023 at 10:46 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > And just to clarify: The voting will last till Tuesday 25 July 20023 > 11am CST. > > > > On Fri, Jul 21, 2023 at 10:43 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > Clarification: Ash is right (I totally forgot about it). > > > > > > We do have dask in > > > > https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html#core-airflow-extras > > > and it even explicitly mentions it: > > > > > > * "dask pip install 'apache-airflow[dask]' DaskExecutor" > > > > > > So technically speaking you **should** use the dask extra to install > > > distributed, dask and cloudpickle package - all needed to run the dask > > > executor. And we will keep it this way - it will just move from "core > > > extras" to "providers". Actually it will move to "deprecated" and we > > > will alias "dask" it with the new "daskexecutor", because the name > > > "apache-airflow-providers-dask" is not allowed by PyPI (I guess the > > > -dask suffix is considered potentially harmful for typosquatting) so I > > > reserved "apache-airflow-providers-daskexecutor". > > > > > > J. > > > > > > > > > > > > On Fri, Jul 21, 2023 at 10:34 AM Hussein Awala <huss...@awala.fr> > wrote: > > > > > > > > > based on the premise that your had to install the `dask` extra in > the > > > > first place to get dask module of the right version > > > > > > > > After reviewing the Dask Executor > > > > < > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html > > > > > > section in Airflow documentation, I didn't find any requirement or > > > > recommendation to add the `dask` extra when using it. So, including > the > > > > dependencies explicitly in the requirements file should suffice to > use the > > > > executor. This means that this change might potentially break > existing > > > > setups for some users when they upgrade Airflow version, which > contradicts > > > > the Airflow deprecation policy. > > > > > > > > If the premise holds true for Dask, it should also apply to Celery > and > > > > Kubernetes. Nevertheless, we decided to pre-install their providers > after > > > > the migration. > > > > > > > > However, this executor isn't as popular as the others, and as an > Airflow > > > > user, I prefer not to have Dask dependencies in my environment when I > > > > install Airflow with the Celery executor. This helps avoid conflicts > and > > > > enables me to upgrade libraries independently from their > constraints. One > > > > possible solution could be allowing users to instruct Airflow not to > > > > pre-install executors providers by passing an argument via > > > > `--install-option` during pip install, and add install them > explicitly by > > > > providing the extras. > > > > > > > > Since the flexibility of our deprecation policy isn't entirely clear > to me, > > > > I'm neutral on this matter, with a vote of +0. > > > > > > > > On Fri, Jul 21, 2023 at 9:10 AM Elad Kalif <elad...@apache.org> > wrote: > > > > > > > > > I agree with Ash > > > > > -1 as well. > > > > > > > > > > > > > > > On Fri, Jul 21, 2023 at 9:29 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > > > > > > > -1 - based on the premise that your had to install the `dask` > extra in > > > > > the > > > > > > first place to get dask module of the right version, so if we > make the > > > > > > existing extra depends on the new provider then it's good enough. > > > > > > > > > > > > On 21 July 2023 06:22:28 BST, Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > >Q: Do we want to pre-install Dask provider in Airflow 2.7.0 > (with Dask > > > > > > >executor) once separated ? > > > > > > > > > > > > > >Discussion was here: > > > > > > > > https://lists.apache.org/thread/0d9x4kl7hc2qzvho2mbdf35ohn5w12l6 > > > > > > > > > > > > > >Please vote: > > > > > > > > > > > > > >* +1 -> yes, we want to have dask provider preinstalled > > > > > > >* -1 -> no, it's fine to make it optional > > > > > > >* 0 -> no opinion > > > > > > > > > > > > > >Consider it my -1: I think we should NOT preinstall Dask > provider. > > > > > > > > > > > > > >Voting guidelines here. This is really a "procedural" matter > rather > > > > > > >than code modification: > > > > > > > > > > > > > >> Votes on procedural issues follow the common format of > majority rule > > > > > > unless otherwise stated. That is, if there are more favourable > votes than > > > > > > unfavourable ones, the issue is considered to have passed -- > regardless > > > > > of > > > > > > the number of votes in each category. (If the number of votes > seems too > > > > > > small to be representative of a community consensus, the issue is > > > > > typically > > > > > > not pursued. > > > > > > > > > > > > > >In this case committers have binding votes but other community > members > > > > > > >are encouraged to state their non-binding votes as well. > > > > > > > > > > > > > >https://www.apache.org/foundation/voting.html > > > > > > > > > > > > > > >--------------------------------------------------------------------- > > > > > > >To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > > >For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
