Re: flink-ml algorithms

Hi, I have another question. Is the implementation of kmeans in flink-ml same as Spark's StreamingKmeans? Should the accuracy/results from the same dataset be comparable between the two? On Sun, Jun 5, 2022 at 8:14 PM Natia Chachkhiani < natia.chachkhia...@gmail.com> wrote: > Thanks for the reply

Re: flink-ml algorithms

Thanks for the reply Zhipeng and Jing. Running the OnlineKmeans with a fixed initial model removed the randomness! On Sun, Jun 5, 2022 at 6:19 PM Zhipeng Zhang wrote: > Hi Natia, > > As I understand, the processing order of onlineKmeans is the same the > input data. > > Are you running OnlineKm

Re: flink-ml algorithms

Hi Natia, As I understand, the processing order of onlineKmeans is the same the input data. Are you running OnlineKmeans with using one data point with random initial KmeansModel? Could you use a fixed initial model following [1] and try out? [1] https://github.com/apache/flink-ml/blob/239788f2b

Re: flink-ml algorithms

Hi, It seems like an evaluation with a small dataset. In this case, would you like to share your data sample and code? In addition, have you tried KMeans with the same dataset and got inconsistent results too? Best regards, Jing On Fri, Jun 3, 2022 at 4:29 AM Natia Chachkhiani < natia.chachkhia.

Re: Flink-ML: Sink model data in online training

Hi thekingofcity, Thanks for your interest! Unfortunately we don't have an example for online learning for now. We are working on an online machine learning example. Hopefully it will be added here [1] in the next three weeks. [1] https://github.com/apache/flink-ml thekingofcity 于2022年1月26日周三

Re: Flink ML

Currently, FLIP-39 is mainly driven by Becket and his team. I'm including him, maybe he can answer your question. Best, Jark On Wed, 17 Jun 2020 at 23:00, Piotr Nowojski wrote: > Hi, > > It looks like FLIP-39 is only partially implemented as for now [1], so I’m > not sure which features are alr

Re: Flink ML

Hi, It looks like FLIP-39 is only partially implemented as for now [1], so I’m not sure which features are already done. I’m including Shaoxuan Wang in this thread, maybe he will be able to better answer your question. Piotrek [1] https://issues.apache.org/jira/browse/FLINK-12470

Re: Flink ML feature

Hi guys, Yes, as Till mentioned. The community is working on a new ML library and we are working closely with the Alink project to bring the algorithms. You can find more information regarding the new ML design architecture in FLIP-39 [1]. One of the major change is that the new ML library [2] wi

Re: Flink ML feature

Hi guys, it is true that we dropped Flink-ML with 1.9. The reason is that the community started working on a new ML library which you can find under flink-ml-parent [1]. This module contains the framework for building ML pipelines but not yet too many algorithms iirc. The plan is to extend this li

Re: Flink ML feature

Hi Benoit, I can only try to ping @Till Rohrmann @Kurt Young who may know more information to answer this question. Best, Vino Benoît Paris 于2019年12月10日周二 下午7:06写道： > Is there any information as to whether Alink is going to be contributed to > Apache Flink as the official ML Lib? > > > On T

Re: Flink ML feature

Is there any information as to whether Alink is going to be contributed to Apache Flink as the official ML Lib? On Tue, Dec 10, 2019 at 7:11 AM vino yang wrote: > Hi Chandu, > > AFAIK, there is a project named Alink[1] which is the Machine Learning > algorithm platform based on Flink, developed

Re: Flink ML feature

Hi Chandu, AFAIK, there is a project named Alink[1] which is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform. FYI Best, Vino [1]: https://github.com/alibaba/Alink Tom Blackwood 于2019年12月10日周二 下午2:07写道： > You may try Spark ML, whi

Re: Flink ML feature

You may try Spark ML, which is a production ready library for ML stuff. regards. On Tue, Dec 10, 2019 at 1:04 PM chandu soa wrote: > Hello Community, > > Can you please give me some pointers for implementing Machine Learning > using Flink. > > I see Flink ML libraries were dropped in v1.9. It l

Re: Flink ML Use cases

Thanks for the confirmation, Fabian. *Regards,* *Abhishek Kumar Singh* *Search Engine Engineer* *Mob :+91 7709735480 * *...* On Sat, May 25, 2019 at 8:55 PM Fabian Hueske wrote: > Hi Abhishek, > > Your observation is correct. Right now, the Flink ML module is in a > half-baked state and is

Re: Flink ML Use cases

Hi Abhishek, Your observation is correct. Right now, the Flink ML module is in a half-baked state and is only supported in batch mode. It is not integrated with the DataStream API. FLIP-23 proposes a feature that allows to evaluated an externally trained model (stored as PMML) on a stream of data.

Re: Flink ML Use cases

Thanks again for the above resources. I went through the project and also ran the example on my system to get a grasp of the architecture. However, this project does not use Flink ML in it at all. Also, after having done enough research on Flink ML, I also found that it does not let us persist t

Re: Flink ML Use cases

Thanks a lot Rong and Sameer. Looks like this is what I wanted. I will try the above projects. *Regards,* *Abhishek Kumar Singh* *Search Engineer* *Mob :+91 7709735480 * *...* On Wed, May 15, 2019 at 8:00 AM Rong Rong wrote: > Hi Abhishek, > > Based on your description, I think this FLIP

Re: Flink ML Use cases

Hi Abhishek, Based on your description, I think this FLIP proposal[1] seems to fit perfectly for your use case. you can also checkout the Github repo by Boris (CCed) for the PMML implementation[2]. This proposal is still under development [3], you are more than welcome to test out and share your f

Re: Flink ML Use cases

If you can save the model as a PMML file you can apply it on a stream using one of the java pmml libraries. Sent from my iPhone > On May 14, 2019, at 4:44 PM, Abhishek Singh wrote: > > I was looking forward to using Flink ML for my project where I think I can > use SVM. > > I have been able

Re: Flink ML with DataStream

could produce something reusable for the community. > > > > > > *From:* Fabian Hueske [mailto:fhue...@gmail.com] > *Sent:* Wednesday, July 19, 2017 2:12 PM > *To:* Branham, Jeremy [IT] > *Cc:* user@flink.apache.org > *Subject:* Re: Flink ML with DataStream > >

RE: Flink ML with DataStream

] Sent: Wednesday, July 19, 2017 2:12 PM To: Branham, Jeremy [IT] Cc: user@flink.apache.org Subject: Re: Flink ML with DataStream Hi, unfortunately, it is not possible to convert a DataStream into a DataSet. Flink's DataSet and DataStream APIs are distinct APIs that cannot be used together.

Re: Flink ML with DataStream

Hi, unfortunately, it is not possible to convert a DataStream into a DataSet. Flink's DataSet and DataStream APIs are distinct APIs that cannot be used together. The FlinkML library is only available for the DataSet API. There is some ongoing work to add a machine learning library for streaming u

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Answering to myself if someone is having similar problems. So already saved matrices can be read and used in als like this: // Setup the ALS learnerd val als = ALS() val users = env.readFile(new TypeSerializerInputFormat[Factors](createTypeInformation[Factors]),"path") val i

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Now I got this working in cloud (not locally, but it's ok) so thanks a lot. Next problem is how to read then these written files and add them to the als. I guess it is something like val als = ALS() als.factorsOption = Option(users,items) but I don't get how I could read in the data I hav

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Sorry, I had a mistake in my example code. I thought the model would be stored as a (Option[DataSet[Factors]], Option[DataSet[Factors]]) but instead it’s stored as Option[(DataSet[Factors], DataSet[Factors])]. So the code should be val als = ALS() als.fit(input) val alsModelOpt = als.factorsOpt

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Hi, those parts were examples how I had tried. I tried with your suggestions, but still no success. Additionally, there were some problems: val (userFactorsOpt, itemFactorsOpt) = als.factorsOption If I had just this, userFactorsOpt And itemFactorsOpt did not have write method. So I added ge

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Hi Kirsti, I think you attached some images to your file which show the code. Unfortunately this is not supported by the mailing list. So maybe you could resend what you’ve already tried. In order to access the ALS model, you can do the following: val als = ALS() als.fit(input) val (userFactor

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

How should this be done for the recommendation engine (that is ALS, example here https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/libs/ml/als.html ). I am able to run the example wit

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

I'm just about to open an issue / PR solution for 'warm-starts' Once this is in, we could just add a setter for the weight vector (and what ever iteration you're on if you're going to do more partial fits). Then all you need to save if your weight vector (and iter number). Trevor Grant Data Sc

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Is there a reasons the Predictor or Estimator class don't have read and write methods for saving and retrieving the model? I couldn't find Jira issues for it. Does it make sense to create one ? BR, Behrouz On Wed, Mar 30, 2016 at 4:40 PM, Till Rohrmann wrote: > Yes Suneel is completely wright.

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Yes Suneel is completely wright. If the data does not implement IOReadableWritable it is probably easier to use the TypeSerializerOutputFormat. What you need here to seralize the data is a TypeSerializer. You can obtain it the following way: val model = mlr.weightsOption.get val weightVectorTypeI

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

U may want to use FlinkMLTools.persist() methods which use TypeSerializerFormat and don't enforce IOReadableWritable. On Tue, Mar 29, 2016 at 2:12 PM, Sourigna Phetsarath < gna.phetsar...@teamaol.com> wrote: > Till, > > Thank you for your reply. > > Having this issue though, WeightVector does n

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Till, Thank you for your reply. Having this issue though, WeightVector does not extend IOReadWriteable: *public* *class* SerializedOutputFormat<*T* *extends* IOReadableWritable> *case* *class* WeightVector(weights: Vector, intercept: Double) *extends* Serializable {} However, I will use the a

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Hi Gna, there are no utilities yet to do that but you can do it manually. In the end, a model is simply a Flink DataSet which you can serialize to some file. Upon reading this DataSet you simply have to give it to your algorithm to be used as the model. The following code snippet illustrates this

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

To my knowledge there is nothing like that. PMML is not supported in any form and there's no custom saving format yet. If you really need a quick and dirty solution, it's not that hard to serialize the model into a file. 2016-03-28 17:59 GMT+02:00 Sourigna Phetsarath : > Flinksters, > > Is there

Re: Flink-ml multiple linear regression fit

You are again missing a library. There seems so be something quite complicated about your build setup. I would go for the ML quickstart or Maven template, which will package a correct fat jar automatically. On Sun, Sep 20, 2015 at 2:15 PM, Florian Heyl wrote: > Hi Stephan, > > Yeah I forgot

Re: Flink-ml multiple linear regression fit

Hi Stephan, Yeah I forgot the breeze library. Thanks. Unfortunately there is still another problem when I am running the pipeline on the hdfs. I tried to figure out what the cause of the problem is and I am mainly stuck at the collect method for the datasets. // List( (1.0, 1.0), (2.0, 2.0), .

Re: Flink-ml multiple linear regression fit

Hi! Looks like you submitted the program JAR, but it did not contain all required libraries, like the breeze JAR. Did you build a proper fat jar, or how did you package the program? Greetings, Stephan On Fri, Sep 18, 2015 at 8:22 PM, Florian Heyl wrote: > Hey Guys need your help again, > I am

Re: Flink-ML as Dependency

Well then, I should update ;-) On Thu, Jun 11, 2015 at 4:01 PM, Till Rohrmann wrote: > Hmm then I assume that version 2 can properly handle maven property > variables. > > > On Thu, Jun 11, 2015 at 3:05 PM Maximilian Alber < > alber.maximil...@gmail.com> wrote: > >> Hi Till, >> >> I use the stan

Re: Flink-ML as Dependency

Hmm then I assume that version 2 can properly handle maven property variables. On Thu, Jun 11, 2015 at 3:05 PM Maximilian Alber wrote: > Hi Till, > > I use the standard one for Ubuntu 15.04, which is 1.5. > > That did not make any difference. > > Thanks and Cheers, > Max > > On Thu, Jun 11, 2015

Re: Flink-ML as Dependency

Hi Till, I use the standard one for Ubuntu 15.04, which is 1.5. That did not make any difference. Thanks and Cheers, Max On Thu, Jun 11, 2015 at 11:22 AM, Till Rohrmann wrote: > Hi Max, > > I just tested a build using gradle (with your build.gradle file) and some > flink-ml algorithms. And it

Re: Flink-ML as Dependency

Hi Max, I just tested a build using gradle (with your build.gradle file) and some flink-ml algorithms. And it was completed without the problem of the unresolved breeze dependency. I use the version 2.2.1 of Gradle. Which version are you using? Since you’re using Flink’s snapshots and have speci

Re: Flink-ML as Dependency

Hi Till, Thanks for the quick help! Cheers, Max On Wed, Jun 10, 2015 at 5:50 PM, Till Rohrmann wrote: > Hi Max, > > I think the reason is that the flink-ml pom contains as a dependency an > artifact with artifactId breeze_${scala.binary.version}. The variable > scala.binary.version is defined

Re: Flink-ML as Dependency

Hi Max, I think the reason is that the flink-ml pom contains as a dependency an artifact with artifactId breeze_${scala.binary.version}. The variable scala.binary.version is defined in the parent pom and not substituted when flink-ml is installed. Therefore gradle tries to find a dependency with t

Re: flink ml - k-means

okay :) now i use the following exsample code from here: https://github.com/apache/flink/blob/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java 2015-05-11 21:56 GMT+02:00 Stephan Ewen : > Paul! > > Can you use the KMeans example? The co

Re: flink ml - k-means

Paul! Can you use the KMeans example? The code is for three-dimensional points, but you should be able to generalize it easily. That would be the fastest way to go. without waiting for any release dates... Stephan On Mon, May 11, 2015 at 2:46 PM, Pa Rö wrote: > hi, > > now i want implement km

Re: flink ml k means relase

Hi, the community didn't decide on a plan for releasing Flink 0.9 yet. Here, you can track the progress for the Flink ML variant of KMeans: https://issues.apache.org/jira/browse/FLINK-1731 There is also a KMeans implementation in the examples of Flink. Maybe that is sufficient for now? --Robert

Re: flink ml - k-means

hi, now i want implement kmeans with flink, maybe you know a release date for flink ml kmeans? best regards paul 2015-04-27 9:36 GMT+02:00 Pa Rö : > Hi Alexander and Till, > > thanks for your informations, I look forward to the release. > I'm curious how well is flink ml against mahout und spar

Re: flink ml - k-means

Hi Alexander and Till, thanks for your informations, I look forward to the release. I'm curious how well is flink ml against mahout und spark ml. best regerds Paul 2015-04-27 9:23 GMT+02:00 Till Rohrmann : > Hi Paul, > > if you can't wait, a vanilla implementation is already contained as part >

Re: flink ml - k-means

Hi Paul, if you can't wait, a vanilla implementation is already contained as part of the Flink examples. You should find it under flink/flink-examples. But we will try to add more clustering algorithms in the near future. Cheers, Till On Apr 26, 2015 11:14 PM, "Alexander Alexandrov" < alexander.

Re: flink ml - k-means

Yes, I expect to have one in the next few weeks (the code is actually there, but we need to port it to the Flink ML API). I suggest to follow the JIRA issue in the next weeks to check when this is done: https://issues.apache.org/jira/browse/FLINK-1731 Regards, Alexander PS. Bear in mind that we