Re: Workflow Scheduler for Spark

I created Jira and design doc on this matter. 2014-09-17 22:28 GMT+04:00 Reynold Xin : > There might've been some misunderstanding. I was referring

Re: Workflow Scheduler for Spark

There might've been some misunderstanding. I was referring to the MLlib pipeline design doc when I said the design doc was posted, in response to the first paragraph of your original email. On Wed, Sep 17, 2014 at 2:47 AM, Egor Pahomov wrote: > It's doc about MLLib pipeline functionality. What

Re: Workflow Scheduler for Spark

It's doc about MLLib pipeline functionality. What about oozie-like workflow? 2014-09-17 13:08 GMT+04:00 Mark Hamstra : > See https://issues.apache.org/jira/browse/SPARK-3530 and this doc, > referenced in that JIRA: > > > https://docs.google.com/document/d/1rVwXRjWKfIb-7PI6b86ipytwbUH7irSNLF1_6dLm

Re: Workflow Scheduler for Spark

See https://issues.apache.org/jira/browse/SPARK-3530 and this doc, referenced in that JIRA: https://docs.google.com/document/d/1rVwXRjWKfIb-7PI6b86ipytwbUH7irSNLF1_6dLmh8o/edit?usp=sharing On Wed, Sep 17, 2014 at 2:00 AM, Egor Pahomov wrote: > I have problems using Oozie. For example it doesn't

Re: Workflow Scheduler for Spark

I have problems using Oozie. For example it doesn't sustain spark context like ooyola job server does. Other than GUI interfaces like HUE it's hard to work with - scoozie stopped in development year ago(I spoke with creator) and oozie xml very hard to write. Oozie still have all documentation and c

Re: Workflow Scheduler for Spark

Hi Egor, I think the design doc for the pipeline feature has been posted. For the workflow, I believe Oozie actually works fine with Spark if you want some external workflow system. Do you have any trouble using that? On Tue, Sep 16, 2014 at 11:45 PM, Egor Pahomov wrote: > There are two thing