+1 for keeping it and fixing tests PS, I also noticed the skipped tests while looking at an option to use the async client feature; if/when I get time to get back on that and figure out how the test setup needs to work, I might also discover how to enable tests for the non-async executor. No promises, just noting that I'm aware of it too.
On Mon, Jan 13, 2020 at 8:06 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > For now I marked the skipped tests we had (including Dask) as > pytest.mark.xfail (means - expected to fail). They will be executed and > summarized as XFail tests and we will have to deal with them at some point. > > I think we will have to decide if we want to keep it or not, and either > remove both tests and executor or fix the tests. > > J. > > On Mon, Jan 13, 2020 at 4:40 PM Shaw, Damian P. < > damian.sha...@credit-suisse.com> wrote: > > > FYI I used Dash instead of Local Executor when first starting Airflow, it > > was a great way to make sure the Executor and Scheduler weren’t tied to > > each other with no difficulty in set-up. But once I actually started > > deploying to multiple boxes I needed queue names pretty quickly. So not > > going to say it's needed but for me it was a helpful stepping stone. > > > > > > -----Original Message----- > > From: Ash Berlin-Taylor <a...@apache.org> > > Sent: Sunday, January 12, 2020 17:38 > > To: dev@airflow.apache.org > > Cc: dev@airflow.apache.org > > Subject: Re: Remove Dask Executor in Airflow 2.0 ? > > > > It hasn't been discussed before, but unlike the Mesos one this one was > > seen a (tiny) bit of activity in 1.10 so at least one person is using it > > https://github.com/apache/airflow/pull/5273 > > > > On Jan 12 2020, at 9:05 pm, Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > I am finishing the PR on separating integrations and improving our CI > > > footprint (https://github.com/apache/airflow/pull/7091) but during > > > this change I have found that we have - apparently - dysfunctional > > > DaskExecutor in Airflow 2.0. > > > > > > There is a "test_dask_executor.py" for which all tests are skipped. > > > And they fail when I try to run the tests. I tried to look for any > > > reference in devlist archives but I couldn't find anything about it. > > > > > > Can someone shed some light on this? Should we remove Dask executor > > > completely from Airflow 2.0 ? Or should we fix the tests/executor ? > > > Has it been discussed ? > > > > > > J. > > > > > > -- > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > =============================================================================== > > > > Please access the attached hyperlink for an important electronic > > communications disclaimer: > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > > =============================================================================== > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Darren L. Weber, Ph.D. http://psdlw.users.sourceforge.net/ http://psdlw.users.sourceforge.net/wordpress/
