Thank you Felix, for your point, it is quite interesting. I will take a look at the code, of the provided Flink integration.
1) You have these problems with Flink: >>we realized that the lack of a caching operator and a broadcast issue highly effects the performance, have you already asked about this the community? In case yes: please provide the reference to the ticket or the topic of letter. 2) You have said, that SystemML provides GPU support. I have seen SystemML’s source code and would like to ask: why you have decided to implement your own integration with cuda? Did you try to consider ND4J, or because it is younger, you support your own implementation? вт, 7 февр. 2017 г. в 18:35, Felix Neutatz <neut...@googlemail.com>: > Hi Katherin, > > we are also working in a similar direction. We implemented a prototype to > integrate with SystemML: > https://github.com/apache/incubator-systemml/pull/119 > SystemML provides many different matrix formats, operations, GPU support > and a couple of DL algorithms. Unfortunately, we realized that the lack of > a caching operator and a broadcast issue highly effects the performance > (e.g. compared to Spark). At the moment I am trying to tackle the broadcast > issue. But caching is still a problem for us. > > Best regards, > Felix > > 2017-02-07 16:22 GMT+01:00 Katherin Eri <katherinm...@gmail.com>: > > > Thank you, Till. > > > > 1) Regarding ND4J, I didn’t know about such a pity and critical > > restriction of it -> lack of sparsity optimizations, and you are right: > > this issue is still actual for them. I saw that Flink uses Breeze, but I > > thought its usage caused by some historical reasons. > > > > 2) Regarding integration with DL4J, I have read the source code of > > DL4J/Spark integration, that’s why I have declined my idea of reuse of > > their word2vec implementation for now, for example. I can perform deeper > > investigation of this topic, if it required. > > > > > > > > So I feel that we have the following picture: > > > > 1) DL integration investigation, could be part of Apache Bahir. I > can > > perform futher investigation of this topic, but I thik we need some > > separated ticket for this to track this activity. > > > > 2) GPU support, required for DL is interesting, but requires ND4J > for > > example. > > > > 3) ND4J couldn’t be incorporated because it doesn’t support sparsity > > <https://deeplearning4j.org/roadmap.html> [1]. > > > > Regarding ND4J is this the single blocker for incorporation of it or may > be > > some others known? > > > > > > [1] https://deeplearning4j.org/roadmap.html > > > > вт, 7 февр. 2017 г. в 16:26, Till Rohrmann <trohrm...@apache.org>: > > > > Thanks for initiating this discussion Katherin. I think you're right that > > in general it does not make sense to reinvent the wheel over and over > > again. Especially if you only have limited resources at hand. So if we > > could integrate Flink with some existing library that would be great. > > > > In the past, however, we couldn't find a good library which provided > enough > > freedom to integrate it with Flink. Especially if you want to have > > distributed and somewhat high-performance implementations of ML > algorithms > > you would have to take Flink's execution model (capabilities as well as > > limitations) into account. That is mainly the reason why we started > > implementing some of the algorithms "natively" on Flink. > > > > If I remember correctly, then the problem with ND4J was and still is that > > it does not support sparse matrices which was a requirement from our > side. > > As far as I know, it is quite common that you have sparse data structures > > when dealing with large scale problems. That's why we built our own > > abstraction which can have different implementations. Currently, the > > default implementation uses Breeze. > > > > I think the support for GPU based operations and the actual resource > > management are two orthogonal things. The implementation would have to > work > > with no GPUs available anyway. If the system detects that GPUs are > > available, then ideally it would exploit them. Thus, we could add this > > feature later and maybe integrate it with FLINK-5131 [1]. > > > > Concerning the integration with DL4J I think that Theo's proposal to do > it > > in a separate repository (maybe as part of Apache Bahir) is a good idea. > > We're currently thinking about outsourcing some of Flink's libraries into > > sub projects. This could also be an option for the DL4J integration then. > > In general I think it should be feasible to run DL4J on Flink given that > it > > also runs on Spark. Have you already looked at it closer? > > > > [1] https://issues.apache.org/jira/browse/FLINK-5131 > > > > Cheers, > > Till > > > > On Tue, Feb 7, 2017 at 11:47 AM, Katherin Eri <katherinm...@gmail.com> > > wrote: > > > > > Thank you Theodore, for your reply. > > > > > > 1) Regarding GPU, your point is clear and I agree with it, ND4J > looks > > > appropriate. But, my current understanding is that, we also need to > cover > > > some resource management questions -> when we need to provide GPU > support > > > we also need to manage it like resource. For example, Mesos has already > > > supported GPU like resource item: Initial support for GPU resources. > > > <https://issues.apache.org/jira/browse/MESOS-4424?jql=text%20~%20GPU> > > > Flink > > > uses Mesos as cluster manager, and this means that this feature of > Mesos > > > could be reused. Also memory managing questions in Flink regarding GPU > > > should be clarified. > > > > > > 2) Regarding integration with DL4J: what stops us to initialize > ticket > > > and start the discussion around this topic? We need some user story or > > the > > > community is not sure that DL is really helpful? Why the discussion > with > > > Adam > > > Gibson just finished with no implementation of any idea? What concerns > do > > > we have? > > > > > > пн, 6 февр. 2017 г. в 15:01, Theodore Vasiloudis < > > > theodoros.vasilou...@gmail.com>: > > > > > > > Hell all, > > > > > > > > This is point that has come up in the past: Given the multitude of ML > > > > libraries out there, should we have native implementations in FlinkML > > or > > > > try to integrate other libraries instead? > > > > > > > > We haven't managed to reach a consensus on this before. My opinion is > > > that > > > > there is definitely value in having ML algorithms written natively in > > > > Flink, both for performance optimization, > > > > but more importantly for engineering simplicity, we don't want to > force > > > > users to use yet another piece of software to run their ML algos (at > > > least > > > > for a basic set of algorithms). > > > > > > > > We have in the past discussed integrations with DL4J (particularly > > ND4J) > > > > with Adam Gibson, the core developer of the library, but we never got > > > > around to implementing anything. > > > > > > > > Whether it makes sense to have an integration with DL4J as part of > the > > > > Flink distribution would be up for discussion. I would suggest to > make > > it > > > > an independent repo to allow for > > > > faster dev/release cycles, and because it wouldn't be directly > related > > to > > > > the core of Flink so it would add extra reviewing burden to an > already > > > > overloaded group of committers. > > > > > > > > Natively supporting GPU calculations in Flink would be much better > > > achieved > > > > through a library like ND4J, the engineering burden would be too much > > > > otherwise. > > > > > > > > Regards, > > > > Theodore > > > > > > > > On Mon, Feb 6, 2017 at 11:26 AM, Katherin Eri < > katherinm...@gmail.com> > > > > wrote: > > > > > > > > > Hello, guys. > > > > > > > > > > Theodore, last week I started the review of the PR: > > > > > https://github.com/apache/flink/pull/2735 related to *word2Vec for > > > > Flink*. > > > > > > > > > > > > > > > > > > > > During this review I have asked myself: why do we need to implement > > > such > > > > a > > > > > very popular algorithm like *word2vec one more time*, when there is > > > > already > > > > > available implementation in java provided by deeplearning4j.org > > > > > <https://deeplearning4j.org/word2vec> library (DL4J -> Apache 2 > > > > licence). > > > > > This library tries to promote itself, there is a hype around it in > ML > > > > > sphere, and it was integrated with Apache Spark, to provide > scalable > > > > > deeplearning calculations. > > > > > > > > > > > > > > > *That's why I thought: could we integrate with this library or not > > also > > > > and > > > > > Flink? * > > > > > > > > > > 1) Personally I think, providing support and deployment of > > > > > *Deeplearning(DL) > > > > > algorithms/models in Flink* is promising and attractive feature, > > > because: > > > > > > > > > > a) during last two years DL proved its efficiency and these > > > > algorithms > > > > > used in many applications. For example *Spotify *uses DL based > > > algorithms > > > > > for music content extraction: Recommending music on Spotify with > deep > > > > > learning AUGUST 05, 2014 > > > > > <http://benanne.github.io/2014/08/05/spotify-cnns.html> for their > > > music > > > > > recommendations. Developers need to scale up DL manually, that > causes > > a > > > > lot > > > > > of work, so that’s why such platforms like Flink should support > these > > > > > models deployment. > > > > > > > > > > b) Here is presented the scope of Deeplearning usage cases > > > > > <https://deeplearning4j.org/use_cases>, so many of this scenarios > > > > related > > > > > to scenarios, that could be supported on Flink. > > > > > > > > > > > > > > > 2) But DL uncover such questions like: > > > > > > > > > > a) scale up calculations over machines > > > > > > > > > > b) perform these calculations both over CPU and GPU. GPU is > > > required > > > > to > > > > > train big DL models, otherwise learning process could have very > slow > > > > > convergence. > > > > > > > > > > > > > > > 3) I have checked this DL4J library, which already have reach > support > > > of > > > > > many attractive DL models like: Recurrent Networks and LSTMs, > > > > Convolutional > > > > > Networks (CNN), Restricted Boltzmann Machines (RBM) and others. So > we > > > > won’t > > > > > need to implement them independently, but only provide the ability > of > > > > > execution of this models over Flink cluster, the quite similar way > > like > > > > it > > > > > was integrated with Apache Spark. > > > > > > > > > > > > > > > Because of all of this I propose: > > > > > > > > > > 1) To create new ticket in Flink’s JIRA for integration of Flink > > > with > > > > > DL4J and decide on which side this integration should be > implemented. > > > > > > > > > > 2) Support natively GPU resources in Flink and allow > calculations > > > over > > > > > them, like that is described in this publication > > > > > https://www.oreilly.com/learning/accelerating-spark- > > > workloads-using-gpus > > > > > > > > > > > > > > > > > > > > *Regarding original issue Implement Word2Vec > > > > > <https://issues.apache.org/jira/browse/FLINK-2094>in Flink, *I > have > > > > > investigated its implementation in DL4J and that implementation of > > > > > integration DL4J with Apache Spark, and got several points: > > > > > > > > > > It seems that idea of building of our own implementation of > word2vec > > in > > > > > Flink not such a bad solution, because: This DL4J was forced to > > > > reimplement > > > > > its original word2Vec over Spark. I have checked the integration of > > > DL4J > > > > > with Spark, and found that it is too strongly coupled with Spark > API, > > > so > > > > > that it is impossible just to take some DL4J API and reuse it, > > instead > > > we > > > > > need to implement independent integration for Flink. > > > > > > > > > > *That’s why we simply finish implementation of current PR > > > > > **independently **from > > > > > integration to DL4J.* > > > > > > > > > > > > > > > > > > > > Could you please provide your opinion regarding my questions and > > > points, > > > > > what do you think about them? > > > > > > > > > > > > > > > > > > > > пн, 6 февр. 2017 г. в 12:51, Katherin Eri <katherinm...@gmail.com > >: > > > > > > > > > > > Sorry, guys I need to finish this letter first. > > > > > > Full version of it will come shortly. > > > > > > > > > > > > пн, 6 февр. 2017 г. в 12:49, Katherin Eri < > katherinm...@gmail.com > > >: > > > > > > > > > > > > Hello, guys. > > > > > > Theodore, last week I started the review of the PR: > > > > > > https://github.com/apache/flink/pull/2735 related to *word2Vec > for > > > > > Flink*. > > > > > > > > > > > > During this review I have asked myself: why do we need to > implement > > > > such > > > > > a > > > > > > very popular algorithm like *word2vec one more time*, when there > is > > > > > > already availabe implementation in java provided by > > > deeplearning4j.org > > > > > > <https://deeplearning4j.org/word2vec> library (DL4J -> Apache 2 > > > > > licence). > > > > > > This library tries to promote it self, there is a hype around it > in > > > ML > > > > > > sphere, and it was integrated with Apache Spark, to provide > > scalable > > > > > > deeplearning calculations. > > > > > > That's why I thought: could we integrate with this library or not > > > also > > > > > and > > > > > > Flink? > > > > > > 1) Personally I think, providing support and deployment of > > > Deeplearning > > > > > > algorithms/models in Flink is promising and attractive feature, > > > > because: > > > > > > a) during last two years deeplearning proved its efficiency > and > > > > this > > > > > > algorithms used in many applications. For example *Spotify *uses > DL > > > > based > > > > > > algorithms for music content extraction: Recommending music on > > > Spotify > > > > > > with deep learning AUGUST 05, 2014 > > > > > > <http://benanne.github.io/2014/08/05/spotify-cnns.html> for > their > > > > music > > > > > > recommendations. Doing this natively scalable is very attractive. > > > > > > > > > > > > > > > > > > I have investigated that implementation of integration DL4J with > > > Apache > > > > > > Spark, and got several points: > > > > > > > > > > > > 1) It seems that idea of building of our own implementation of > > > word2vec > > > > > > not such a bad solution, because the integration of DL4J with > Spark > > > is > > > > > too > > > > > > strongly coupled with Saprk API and it will take time from the > side > > > of > > > > > DL4J > > > > > > to adopt this integration to Flink. Also I have expected that we > > will > > > > be > > > > > > able to call just some API, it is not such thing. > > > > > > 2) > > > > > > > > > > > > https://deeplearning4j.org/use_cases > > > > > > https://www.analyticsvidhya.com/blog/2017/01/t-sne- > > > > > implementation-r-python/ > > > > > > > > > > > > > > > > > > чт, 19 янв. 2017 г. в 13:29, Till Rohrmann <trohrm...@apache.org > >: > > > > > > > > > > > > Hi Katherin, > > > > > > > > > > > > welcome to the Flink community. Always great to see new people > > > joining > > > > > the > > > > > > community :-) > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Tue, Jan 17, 2017 at 1:02 PM, Katherin Sotenko < > > > > > katherinm...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > ok, I've got it. > > > > > > > I will take a look at > https://github.com/apache/flink/pull/2735 > > . > > > > > > > > > > > > > > вт, 17 янв. 2017 г. в 14:36, Theodore Vasiloudis < > > > > > > > theodoros.vasilou...@gmail.com>: > > > > > > > > > > > > > > > Hello Katherin, > > > > > > > > > > > > > > > > Welcome to the Flink community! > > > > > > > > > > > > > > > > The ML component definitely needs a lot of work you are > > correct, > > > we > > > > > are > > > > > > > > facing similar problems to CEP, which we'll hopefully resolve > > > with > > > > > the > > > > > > > > restructuring Stephan has mentioned in that thread. > > > > > > > > > > > > > > > > If you'd like to help out with PRs we have many open, one I > > have > > > > > > started > > > > > > > > reviewing but got side-tracked is the Word2Vec one [1]. > > > > > > > > > > > > > > > > Best, > > > > > > > > Theodore > > > > > > > > > > > > > > > > [1] https://github.com/apache/flink/pull/2735 > > > > > > > > > > > > > > > > On Tue, Jan 17, 2017 at 12:17 PM, Fabian Hueske < > > > fhue...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Katherin, > > > > > > > > > > > > > > > > > > welcome to the Flink community! > > > > > > > > > Help with reviewing PRs is always very welcome and a great > > way > > > to > > > > > > > > > contribute. > > > > > > > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-01-17 11:17 GMT+01:00 Katherin Sotenko < > > > > > katherinm...@gmail.com > > > > > > >: > > > > > > > > > > > > > > > > > > > Thank you, Timo. > > > > > > > > > > I have started the analysis of the topic. > > > > > > > > > > And if it necessary, I will try to perform the review of > > > other > > > > > > pulls) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 17 янв. 2017 г. в 13:09, Timo Walther < > > > twal...@apache.org > > > > >: > > > > > > > > > > > > > > > > > > > > > Hi Katherin, > > > > > > > > > > > > > > > > > > > > > > great to hear that you would like to contribute! > Welcome! > > > > > > > > > > > > > > > > > > > > > > I gave you contributor permissions. You can now assign > > > issues > > > > > to > > > > > > > > > > > yourself. I assigned FLINK-1750 to you. > > > > > > > > > > > Right now there are many open ML pull requests, you are > > > very > > > > > > > welcome > > > > > > > > to > > > > > > > > > > > review the code of others, too. > > > > > > > > > > > > > > > > > > > > > > Timo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 17/01/17 um 10:39 schrieb Katherin Sotenko: > > > > > > > > > > > > Hello, All! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm Kate Eri, I'm java developer with 6-year > enterprise > > > > > > > experience, > > > > > > > > > > also > > > > > > > > > > > I > > > > > > > > > > > > have some expertise with scala (half of the year). > > > > > > > > > > > > > > > > > > > > > > > > Last 2 years I have participated in several BigData > > > > projects > > > > > > that > > > > > > > > > were > > > > > > > > > > > > related to Machine Learning (Time series analysis, > > > > > Recommender > > > > > > > > > systems, > > > > > > > > > > > > Social networking) and ETL. I have experience with > > > Hadoop, > > > > > > Apache > > > > > > > > > Spark > > > > > > > > > > > and > > > > > > > > > > > > Hive. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m fond of ML topic, and I see that Flink project > > > requires > > > > > > some > > > > > > > > work > > > > > > > > > > in > > > > > > > > > > > > this area, so that’s why I would like to join Flink > and > > > ask > > > > > me > > > > > > to > > > > > > > > > grant > > > > > > > > > > > the > > > > > > > > > > > > assignment of the ticket > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1750 > > > > > > > > > > > > to me. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
