Do we have anyone here who uses Dask Executor and would like to test it/fix the tests. They are marked now as xfailed (expected to fail) and it would be great to fix them.
J. On Tue, Jan 14, 2020 at 12:18 AM Darren Weber <dweber.consult...@gmail.com> wrote: > +1 for keeping it and fixing tests > > PS, I also noticed the skipped tests while looking at an option to use the > async client feature; if/when I get time to get back on that and figure out > how the test setup needs to work, I might also discover how to enable tests > for the non-async executor. No promises, just noting that I'm aware of it > too. > > On Mon, Jan 13, 2020 at 8:06 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > For now I marked the skipped tests we had (including Dask) as > > pytest.mark.xfail (means - expected to fail). They will be executed and > > summarized as XFail tests and we will have to deal with them at some > point. > > > > I think we will have to decide if we want to keep it or not, and either > > remove both tests and executor or fix the tests. > > > > J. > > > > On Mon, Jan 13, 2020 at 4:40 PM Shaw, Damian P. < > > damian.sha...@credit-suisse.com> wrote: > > > > > FYI I used Dash instead of Local Executor when first starting Airflow, > it > > > was a great way to make sure the Executor and Scheduler weren’t tied to > > > each other with no difficulty in set-up. But once I actually started > > > deploying to multiple boxes I needed queue names pretty quickly. So not > > > going to say it's needed but for me it was a helpful stepping stone. > > > > > > > > > -----Original Message----- > > > From: Ash Berlin-Taylor <a...@apache.org> > > > Sent: Sunday, January 12, 2020 17:38 > > > To: dev@airflow.apache.org > > > Cc: dev@airflow.apache.org > > > Subject: Re: Remove Dask Executor in Airflow 2.0 ? > > > > > > It hasn't been discussed before, but unlike the Mesos one this one was > > > seen a (tiny) bit of activity in 1.10 so at least one person is using > it > > > https://github.com/apache/airflow/pull/5273 > > > > > > On Jan 12 2020, at 9:05 pm, Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > I am finishing the PR on separating integrations and improving our CI > > > > footprint (https://github.com/apache/airflow/pull/7091) but during > > > > this change I have found that we have - apparently - dysfunctional > > > > DaskExecutor in Airflow 2.0. > > > > > > > > There is a "test_dask_executor.py" for which all tests are skipped. > > > > And they fail when I try to run the tests. I tried to look for any > > > > reference in devlist archives but I couldn't find anything about it. > > > > > > > > Can someone shed some light on this? Should we remove Dask executor > > > > completely from Airflow 2.0 ? Or should we fix the tests/executor ? > > > > Has it been discussed ? > > > > > > > > J. > > > > > > > > -- > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > =============================================================================== > > > > > > Please access the attached hyperlink for an important electronic > > > communications disclaimer: > > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > > > > > =============================================================================== > > > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > -- > Darren L. Weber, Ph.D. > http://psdlw.users.sourceforge.net/ > http://psdlw.users.sourceforge.net/wordpress/ > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
