Opening a discussion on FlinkML

2016-02-12 Thread Theodore Vasiloudis
Hello all, I would like to get a conversation started on how we plan to move forward with FlinkML. Development on the library currently has been mostly dormant for the past 6 months, mainly I believe because of the lack of available committers to review PRs. Last month we got together with Till

Re: Opening a discussion on FlinkML

2016-02-12 Thread Simone Robutti
I will say my opinion as a person that have worked with SparkML and will be involved soon in the development of ML solutions on Flink. In these days I tried to track the evolution and development of FlinkML and I see a big critical point: FlinkML looks a lot like a placeholder for commercial purpo

Re: Opening a discussion on FlinkML

2016-02-12 Thread Fabian Hueske
Hi Theo, thanks for starting this discussion. You are certainly right that the development of FlinkML is stalling. On the other hand, we regularly see people on the mailing list asking for feature. Regarding your proposed ways to proceed: 1) I am not sure how much it would help to move FlinkML t

[ANNOUNCE] Flink 0.10.2 Released

2016-02-12 Thread Ufuk Celebi
The Flink PMC is pleased to announce the availability of Flink 0.10.2. On behalf of the Flink PMC, I would like to thank everybody who contributed to the release. The official release announcement: http://flink.apache.org/news/2016/02/11/release-0.10.2.html Release binaries: http://apache.openmi

Re: Opening a discussion on FlinkML

2016-02-12 Thread Chiwan Park
Hi, I agree what Theo said. Currently, only few committers spend time to review PRs about FlinkML. But I also agree Fabian’s opinion. I would like to keep FlinkML under main repository of Flink. I hope new committers spending time for FlinkML. About Simone’s opinion, yes, FlinkML is still immat

Re: [ANNOUNCE] Flink 0.10.2 Released

2016-02-12 Thread Fabian Hueske
Thanks Ufuk! 2016-02-12 12:57 GMT+01:00 Ufuk Celebi : > The Flink PMC is pleased to announce the availability of Flink 0.10.2. > > On behalf of the Flink PMC, I would like to thank everybody who contributed > to the release. > > The official release announcement: > http://flink.apache.org/news/20

Re: [ANNOUNCE] Flink 0.10.2 Released

2016-02-12 Thread Maximilian Michels
Bravo! Thank you Ufuk for managing the release! On Fri, Feb 12, 2016 at 2:02 PM, Fabian Hueske wrote: > Thanks Ufuk! > > 2016-02-12 12:57 GMT+01:00 Ufuk Celebi : > >> The Flink PMC is pleased to announce the availability of Flink 0.10.2. >> >> On behalf of the Flink PMC, I would like to thank eve

Re: [ANNOUNCE] Flink 0.10.2 Released

2016-02-12 Thread Stephan Ewen
Good work, thanks! On Fri, Feb 12, 2016 at 2:13 PM, Maximilian Michels wrote: > Bravo! Thank you Ufuk for managing the release! > > On Fri, Feb 12, 2016 at 2:02 PM, Fabian Hueske wrote: > > Thanks Ufuk! > > > > 2016-02-12 12:57 GMT+01:00 Ufuk Celebi : > > > >> The Flink PMC is pleased to announ

Re: [ANNOUNCE] Flink 0.10.2 Released

2016-02-12 Thread Robert Metzger
Thank you for doing a release Ufuk! I just tweeted about it: https://twitter.com/ApacheFlink/status/698130110709428224 On Fri, Feb 12, 2016 at 2:13 PM, Maximilian Michels wrote: > Bravo! Thank you Ufuk for managing the release! > > On Fri, Feb 12, 2016 at 2:02 PM, Fabian Hueske wrote: > > Tha

Re: Opening a discussion on FlinkML

2016-02-12 Thread Theodore Vasiloudis
I think Simone raises some good points here. The truth is that FlinkML is still in its infancy and it will be hard to compete with mllib, H2O and Graphlab in terms of features and algorithm "coverage". My hope has always been that the library will be focused on what Flink does well and implement

[jira] [Created] (FLINK-3393) ExternalProcessRunner wait to finish copying error stream

2016-02-12 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-3393: - Summary: ExternalProcessRunner wait to finish copying error stream Key: FLINK-3393 URL: https://issues.apache.org/jira/browse/FLINK-3393 Project: Flink Issue Type:

FlinkML NeuralNetworks

2016-02-12 Thread Trevor Grant
Hey all, I had a post a while ago about needing neural networks. We specifically need a very special type that are good for time series/sensors called LSTM. We had a talk about pros/cons of using deeplearning4j for this use case and eventually decided it made more sense to implement in native Fl

Re: Opening a discussion on FlinkML

2016-02-12 Thread Slim Baltagi
Hi Meanwhile until FlinkML matures, it might be worth having Flink as the engine powering H2O in a similar way Spark are doing with their Sparkling Water. Any thoughts? Thanks Slim Baltagi On Feb 12, 2016, at 7:25 AM, Theodore Vasiloudis wrote: > I think Simone raises some good points here.

Re: FlinkML NeuralNetworks

2016-02-12 Thread Suneel Marthi
On Fri, Feb 12, 2016 at 8:45 AM, Trevor Grant wrote: > Hey all, > > I had a post a while ago about needing neural networks. We specifically > need a very special type that are good for time series/sensors called > LSTM. We had a talk about pros/cons of using deeplearning4j for this use > case a

Re: FlinkML NeuralNetworks

2016-02-12 Thread Trevor Grant
Agreed. Our reasoning for for contributing straight to Flink was we plan on doing a lot of wierd monkey-ing around with these things, and were going to have to get our hands dirty with some code eventually anyway. The LSTM isn't *that* difficult to implement, and it seems easier to write our own t

Re: FlinkML NeuralNetworks

2016-02-12 Thread Simone Robutti
Asking as someone that never did NN on Flink, would you implement it using JCuda? And would you implement it with model parallelization? Is there any theoretical limit to implement "model and data parallelism" in Flink? If you don't use GPUs and you don't parallelize models and data at the same tim

Re: [ANNOUNCE] Flink 0.10.2 Released

2016-02-12 Thread Till Rohrmann
Thanks for being our release manager Ufuk :-) Great work! On Fri, Feb 12, 2016 at 2:15 PM, Robert Metzger wrote: > Thank you for doing a release Ufuk! > > I just tweeted about it: > https://twitter.com/ApacheFlink/status/698130110709428224 > > > On Fri, Feb 12, 2016 at 2:13 PM, Maximilian Michel

Re: Opening a discussion on FlinkML

2016-02-12 Thread Suneel Marthi
My 2 cents as someone who's done ML over the years - having worked on Oryx 2.0 and Mahout and having used Spark MlLib (read as "had no choice due to strict workplace enforcement") and understands well their limitations. 1. FlinkML in its present form seems like "do it like how Spark did it". 2. T

Re: Opening a discussion on FlinkML

2016-02-12 Thread Simone Robutti
@Suneel 1) Totally agree, as I wrote before. 2)I agree that support for PMML is premature but we shouldn't underestimate the variety and complexity of the uses of ML models in the industry. The adoption of Flink, hopefully, will grow and reach less innovative realities where Random Forests and SV

Re: Opening a discussion on FlinkML

2016-02-12 Thread Suneel Marthi
On Fri, Feb 12, 2016 at 9:40 AM, Simone Robutti < simone.robu...@radicalbit.io> wrote: > @Suneel > > 1) Totally agree, as I wrote before. > > 2)I agree that support for PMML is premature but we shouldn't underestimate > the variety and complexity of the uses of ML models in the industry. The > ado

Re: FlinkML NeuralNetworks

2016-02-12 Thread Trevor Grant
JCuda: No, I'm not willing to rely on servers having NVidia cards (some one who is more familiar with server hardware may correct me, in which case I'll say, "No, because *my* servers don't have NVidia cards- someone else can add"). Paralleization: Yes.Admittedly, very clever use of Python could p

[jira] [Created] (FLINK-3394) Clear up the contract of MutableObjectIterator.next(reuse)

2016-02-12 Thread Gabor Gevay (JIRA)
Gabor Gevay created FLINK-3394: -- Summary: Clear up the contract of MutableObjectIterator.next(reuse) Key: FLINK-3394 URL: https://issues.apache.org/jira/browse/FLINK-3394 Project: Flink Issue Ty

[jira] [Created] (FLINK-3395) Polishing the web UI

2016-02-12 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-3395: - Summary: Polishing the web UI Key: FLINK-3395 URL: https://issues.apache.org/jira/browse/FLINK-3395 Project: Flink Issue Type: Improvement Components: We

Re: [ANNOUNCE] Flink 0.10.2 Released

2016-02-12 Thread Kostas Kloudas
Yes thanks a lot Ufuk! > On Feb 12, 2016, at 3:09 PM, Till Rohrmann wrote: > > Thanks for being our release manager Ufuk :-) Great work! > > On Fri, Feb 12, 2016 at 2:15 PM, Robert Metzger wrote: > >> Thank you for doing a release Ufuk! >> >> I just tweeted about it: >> https://twitter.com/A

[jira] [Created] (FLINK-3396) Job submission Savepoint restore logic flawed

2016-02-12 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-3396: -- Summary: Job submission Savepoint restore logic flawed Key: FLINK-3396 URL: https://issues.apache.org/jira/browse/FLINK-3396 Project: Flink Issue Type: Bug