Maybe we can reach out to a company that does Dask as a service? via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Jan 17, 2020 at 9:31 AM, Jarek Potiuk <jarek.pot...@polidea.com> wrote: Yeah. I think if we do not find anyone willing to champion it (no matter committer or contributor), I would be for dropping it.
J. On Fri, Jan 17, 2020 at 6:07 PM Daniel Imberman <daniel.imber...@gmail.com> wrote: > I think we need to ask “who is going to champion this executor.” I see > that it is being used (a bit), but am concerned if no one with knowledge of > this executor is willing to maintain it. > > I’ve personally never used Dask and the DaskExecutor isn’t super high on > my priority list compared to things like autoscaling, DAG serialization, > etc. > > via Newton Mail [ > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 > ] > On Fri, Jan 17, 2020 at 6:07 AM, Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > Do we have anyone here who uses Dask Executor and would like to test it/fix > the tests. They are marked now as xfailed (expected to fail) and it would > be great to fix them. > > J. > > > On Tue, Jan 14, 2020 at 12:18 AM Darren Weber <dweber.consult...@gmail.com > > > wrote: > > > +1 for keeping it and fixing tests > > > > PS, I also noticed the skipped tests while looking at an option to use > the > > async client feature; if/when I get time to get back on that and figure > out > > how the test setup needs to work, I might also discover how to enable > tests > > for the non-async executor. No promises, just noting that I'm aware of it > > too. > > > > On Mon, Jan 13, 2020 at 8:06 AM Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > For now I marked the skipped tests we had (including Dask) as > > > pytest.mark.xfail (means - expected to fail). They will be executed and > > > summarized as XFail tests and we will have to deal with them at some > > point. > > > > > > I think we will have to decide if we want to keep it or not, and either > > > remove both tests and executor or fix the tests. > > > > > > J. > > > > > > On Mon, Jan 13, 2020 at 4:40 PM Shaw, Damian P. < > > > damian.sha...@credit-suisse.com> wrote: > > > > > > > FYI I used Dash instead of Local Executor when first starting > Airflow, > > it > > > > was a great way to make sure the Executor and Scheduler weren’t tied > to > > > > each other with no difficulty in set-up. But once I actually started > > > > deploying to multiple boxes I needed queue names pretty quickly. So > not > > > > going to say it's needed but for me it was a helpful stepping stone. > > > > > > > > > > > > -----Original Message----- > > > > From: Ash Berlin-Taylor <a...@apache.org> > > > > Sent: Sunday, January 12, 2020 17:38 > > > > To: dev@airflow.apache.org > > > > Cc: dev@airflow.apache.org > > > > Subject: Re: Remove Dask Executor in Airflow 2.0 ? > > > > > > > > It hasn't been discussed before, but unlike the Mesos one this one > was > > > > seen a (tiny) bit of activity in 1.10 so at least one person is using > > it > > > > https://github.com/apache/airflow/pull/5273 > > > > > > > > On Jan 12 2020, at 9:05 pm, Jarek Potiuk <jarek.pot...@polidea.com> > > > wrote: > > > > > I am finishing the PR on separating integrations and improving our > CI > > > > > footprint (https://github.com/apache/airflow/pull/7091) but during > > > > > this change I have found that we have - apparently - dysfunctional > > > > > DaskExecutor in Airflow 2.0. > > > > > > > > > > There is a "test_dask_executor.py" for which all tests are skipped. > > > > > And they fail when I try to run the tests. I tried to look for any > > > > > reference in devlist archives but I couldn't find anything about > it. > > > > > > > > > > Can someone shed some light on this? Should we remove Dask executor > > > > > completely from Airflow 2.0 ? Or should we fix the tests/executor ? > > > > > Has it been discussed ? > > > > > > > > > > J. > > > > > > > > > > -- > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > =============================================================================== > > > > > > > > Please access the attached hyperlink for an important electronic > > > > communications disclaimer: > > > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > > > > > > > > > =============================================================================== > > > > > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > -- > > Darren L. Weber, Ph.D. > > http://psdlw.users.sourceforge.net/ > > http://psdlw.users.sourceforge.net/wordpress/ > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
