> One possible solution could be allowing users to instruct Airflow not to pre-install executors providers by passing an argument via `--install-option` during pip install, and add install them explicitly by providing the extras.
Yes it would be nice. Unfortunately the dependency mechanisms of Python (and `pip`) are not as flexible. You cannot individually remove requirements, and having requirements is the only way you can have "install also that other package" feature (extras is the mechanism to install optional dependencies, but it's not something we should ask regular users to do On Fri, Jul 21, 2023 at 10:46 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > And just to clarify: The voting will last till Tuesday 25 July 20023 11am > CST. > > On Fri, Jul 21, 2023 at 10:43 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > Clarification: Ash is right (I totally forgot about it). > > > > We do have dask in > > https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html#core-airflow-extras > > and it even explicitly mentions it: > > > > * "dask pip install 'apache-airflow[dask]' DaskExecutor" > > > > So technically speaking you **should** use the dask extra to install > > distributed, dask and cloudpickle package - all needed to run the dask > > executor. And we will keep it this way - it will just move from "core > > extras" to "providers". Actually it will move to "deprecated" and we > > will alias "dask" it with the new "daskexecutor", because the name > > "apache-airflow-providers-dask" is not allowed by PyPI (I guess the > > -dask suffix is considered potentially harmful for typosquatting) so I > > reserved "apache-airflow-providers-daskexecutor". > > > > J. > > > > > > > > On Fri, Jul 21, 2023 at 10:34 AM Hussein Awala <huss...@awala.fr> wrote: > > > > > > > based on the premise that your had to install the `dask` extra in the > > > first place to get dask module of the right version > > > > > > After reviewing the Dask Executor > > > <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html> > > > section in Airflow documentation, I didn't find any requirement or > > > recommendation to add the `dask` extra when using it. So, including the > > > dependencies explicitly in the requirements file should suffice to use the > > > executor. This means that this change might potentially break existing > > > setups for some users when they upgrade Airflow version, which contradicts > > > the Airflow deprecation policy. > > > > > > If the premise holds true for Dask, it should also apply to Celery and > > > Kubernetes. Nevertheless, we decided to pre-install their providers after > > > the migration. > > > > > > However, this executor isn't as popular as the others, and as an Airflow > > > user, I prefer not to have Dask dependencies in my environment when I > > > install Airflow with the Celery executor. This helps avoid conflicts and > > > enables me to upgrade libraries independently from their constraints. One > > > possible solution could be allowing users to instruct Airflow not to > > > pre-install executors providers by passing an argument via > > > `--install-option` during pip install, and add install them explicitly by > > > providing the extras. > > > > > > Since the flexibility of our deprecation policy isn't entirely clear to > > > me, > > > I'm neutral on this matter, with a vote of +0. > > > > > > On Fri, Jul 21, 2023 at 9:10 AM Elad Kalif <elad...@apache.org> wrote: > > > > > > > I agree with Ash > > > > -1 as well. > > > > > > > > > > > > On Fri, Jul 21, 2023 at 9:29 AM Ash Berlin-Taylor <a...@apache.org> > > > > wrote: > > > > > > > > > -1 - based on the premise that your had to install the `dask` extra in > > > > the > > > > > first place to get dask module of the right version, so if we make the > > > > > existing extra depends on the new provider then it's good enough. > > > > > > > > > > On 21 July 2023 06:22:28 BST, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > >Q: Do we want to pre-install Dask provider in Airflow 2.7.0 (with > > > > > >Dask > > > > > >executor) once separated ? > > > > > > > > > > > >Discussion was here: > > > > > >https://lists.apache.org/thread/0d9x4kl7hc2qzvho2mbdf35ohn5w12l6 > > > > > > > > > > > >Please vote: > > > > > > > > > > > >* +1 -> yes, we want to have dask provider preinstalled > > > > > >* -1 -> no, it's fine to make it optional > > > > > >* 0 -> no opinion > > > > > > > > > > > >Consider it my -1: I think we should NOT preinstall Dask provider. > > > > > > > > > > > >Voting guidelines here. This is really a "procedural" matter rather > > > > > >than code modification: > > > > > > > > > > > >> Votes on procedural issues follow the common format of majority > > > > > >> rule > > > > > unless otherwise stated. That is, if there are more favourable votes > > > > > than > > > > > unfavourable ones, the issue is considered to have passed -- > > > > > regardless > > > > of > > > > > the number of votes in each category. (If the number of votes seems > > > > > too > > > > > small to be representative of a community consensus, the issue is > > > > typically > > > > > not pursued. > > > > > > > > > > > >In this case committers have binding votes but other community > > > > > >members > > > > > >are encouraged to state their non-binding votes as well. > > > > > > > > > > > >https://www.apache.org/foundation/voting.html > > > > > > > > > > > >--------------------------------------------------------------------- > > > > > >To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > >For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
