There might've been some misunderstanding. I was referring to the MLlib pipeline design doc when I said the design doc was posted, in response to the first paragraph of your original email.
On Wed, Sep 17, 2014 at 2:47 AM, Egor Pahomov <pahomov.e...@gmail.com> wrote: > It's doc about MLLib pipeline functionality. What about oozie-like > workflow? > > 2014-09-17 13:08 GMT+04:00 Mark Hamstra <m...@clearstorydata.com>: > > > See https://issues.apache.org/jira/browse/SPARK-3530 and this doc, > > referenced in that JIRA: > > > > > > > https://docs.google.com/document/d/1rVwXRjWKfIb-7PI6b86ipytwbUH7irSNLF1_6dLmh8o/edit?usp=sharing > > > > On Wed, Sep 17, 2014 at 2:00 AM, Egor Pahomov <pahomov.e...@gmail.com> > > wrote: > > > >> I have problems using Oozie. For example it doesn't sustain spark > context > >> like ooyola job server does. Other than GUI interfaces like HUE it's > hard > >> to work with - scoozie stopped in development year ago(I spoke with > >> creator) and oozie xml very hard to write. > >> Oozie still have all documentation and code in MR model rather than in > >> yarn > >> model. And based on it's current speed of development I can't expect > >> radical changes in nearest future. There is no "Databricks" for oozie, > >> which would have people on salary to develop this kind of radical > changes. > >> It's dinosaur. > >> > >> Reunold, can you help finding this doc? Do you mean just pipelining > spark > >> code or additional logic of persistence tasks, job server, task retry, > >> data > >> availability and extra? > >> > >> > >> 2014-09-17 11:21 GMT+04:00 Reynold Xin <r...@databricks.com>: > >> > >> > Hi Egor, > >> > > >> > I think the design doc for the pipeline feature has been posted. > >> > > >> > For the workflow, I believe Oozie actually works fine with Spark if > you > >> > want some external workflow system. Do you have any trouble using > that? > >> > > >> > > >> > On Tue, Sep 16, 2014 at 11:45 PM, Egor Pahomov < > pahomov.e...@gmail.com> > >> > wrote: > >> > > >> >> There are two things we(Yandex) miss in Spark: MLlib good > abstractions > >> and > >> >> good workflow job scheduler. From threads "Adding abstraction in > MlLib" > >> >> and > >> >> "[mllib] State of Multi-Model training" I got the idea, that > databricks > >> >> working on it and we should wait until first post doc, which would > lead > >> >> us. > >> >> What about workflow scheduler? Is there anyone already working on it? > >> Does > >> >> anyone have a plan on doing it? > >> >> > >> >> P.S. We thought that MLlib abstractions about multiple algorithms run > >> with > >> >> same data would need such scheduler, which would rerun algorithm in > >> case > >> >> of > >> >> failure. I understand, that spark provide fault tolerance out of the > >> box, > >> >> but we found some "Ooozie-like" scheduler more reliable for such long > >> >> living workflows. > >> >> > >> >> -- > >> >> > >> >> > >> >> > >> >> *Sincerely yoursEgor PakhomovScala Developer, Yandex* > >> >> > >> > > >> > > >> > >> > >> -- > >> > >> > >> > >> *Sincerely yoursEgor PakhomovScala Developer, Yandex* > >> > > > > > > > -- > > > > *Sincerely yoursEgor PakhomovScala Developer, Yandex* >
