Error when using FlinkML iterations with KeyedCoProcessFunction

Hi, As the DataStream API's iterativeStream method has been deprecated for future flink releases, the documentation recommend’s using Flink ML's iteration as an alternative. I am trying to build my understanding of the new iterations API as it will be a requirement for our future projects. As

Re: FlinkML 'DenseVector' object has no attribute 'get_fields_by_names'

Thanks for the answer, I'll try. Are there examples or tutorials somewhere on how to use FlinkML in real-life scenarios, such as streaming Kafka through a model? От: Xin Jiang Отправлено: 19 сентября 2023 г. 8:07:11 Кому: Evgeniy Lyutikov Копия:

Re: FlinkML 'DenseVector' object has no attribute 'get_fields_by_names'

Hi Evgeniy, Yes, the reason of the exception is that you are returning an incorrect data type. Flink ML doesn’t have a data type for `DenseVector` but it provides a function called `pyflink.ml.functions.array_to_vector` which returns an `Expression`. So maybe you can modify your UDF to union mu

FlinkML 'DenseVector' object has no attribute 'get_fields_by_names'

Hello community! I'm trying to use FlinkML to train a model on data from a PostgreSQL table and I get an error when I try to view the output table after model AttributeError: 'DenseVector' object has no attribute 'get_fields_by_names' My code: # Create train sourc

FlinkMl

hello, For my master thesis i am comparing ml frameworks on data streams. What is the current status on FlinkML? Is distributed learning possible on multiple nodes? If yes, how? I played around with FlinkML a bit and modeled a simple pipeline for sentiment analysis on tweets. For this I used

Re: info about flinkml

Hi The flinkML has been choosen to drop since Flink-1.9 [1] and a new machine learning library has been developed under the umbrella of FLIP-39 [2][3]. As far as I know, the new Flink ml library has not been completed and you could try Alink [4], a Machine Learning algorithm platform based on

info about flinkml

Hi i m evaluating to adopt flink instead spark for data mining processor. I knew flinkML for this scope but in the last release i cant find it. Why? Can you suggest the best way ? -- Cristian Lorenzetto Direzione ICT e Agenda Digitale U.O. Demand, Progettazione e Sviluppo Software Tel: 041

Re: FlinkML status

Hi Mohamed, the development of FlinkML has been stopped in favour of a new machine learning library which you can find here [1]. Be aware that this library is still under development. [1] https://github.com/apache/flink/tree/master/flink-ml-parent Cheers, Till On Sat, Aug 1, 2020 at 10:35 AM

FlinkML status

Hi, What's the current status of FlinkML? is it still part of Flink? the last Flink release that has documentation about it is 1.8. Thanks, M. Haseeb

flinkML for production

Hello, Is there any production practice on using flinkML for machine learning? If so, where is the link? Thanks.

Re: FlinkML

Hi, You could try to plug MOA/Weka library too. I did some preliminary work with that: https://moa.cms.waikato.ac.nz/moa-with-apache-flink/ but then it is not anymore FlinkML algorithms. Best regards, Christophe 2018-04-18 21:13 GMT+02:00 shashank734 : > There are no active discussions

Re: FlinkML

There are no active discussions or guide on that. But I found this example in the repo : https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/ml/IncrementalLearningSkeleton.java

Re: FlinkML

to build FlinkML streaming job not a > batch job. Examples on https://ci.apache.org/projects > /flink/flink-docs-release-1.4/dev/libs/ml/ are only batch examples. > > Is there any possibility? > > > Best regards. > > -- Christophe

FlinkML

Hi, i wonder if there are possibility to build FlinkML streaming job not a batch job. Examples on https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/ml/ are only batch examples. Is there any possibility? Best regards.

Running FlinkML ALS with more than two features

Hello Flink experts, I am new to FlinkML and currently playing around with using ALS in a recommender system. In our dataset, we have more than 2 features. When I tried running the example towards the bottom of this page: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/ml

Re: FlinkML ALS is taking too long to run

th ALS as well at the time > and I recorded improvements by increasing the number of blocks / decreasing > #TSs/TM like you've stated out. > > Cheers, > > Andrea > > > > > > > -- > View this message in context: http://apache-flink-user- > mailing

Re: FlinkML ALS is taking too long to run

-list-archive.2336050.n4.nabble.com/FlinkML-ALS-is-taking-too-long-to-run-tp14154p14192.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: FlinkML ALS is taking too long to run

context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkML-ALS-is-taking-too-long-to-run-tp14154p14186.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

FlinkML ALS is taking too long to run

Dear all I'm trying to run Flink ALS against Yahoo-R2 data set[1] on HDFS. The program is running without showing any errors, but it does not finish. The operators running indefinitely are: CoGroup (CoGroup at org.apache.flink.ml.recommendation.ALS$.updateFactors(ALS.scala:606))(11/240) Join(Joi

Re: Using FlinkML from Java?

Hi Steve, unfortunately, FlinkML's pipeline mechanism depends on Scala's implicit value feature. Therefore, FlinkML can only be used with Scala if you don't want to construct the pipelines manually (which I wouldn't recommend). Cheers, Till On Thu, Apr 20, 2017 at 6:56 PM,

Using FlinkML from Java?

Hi Folks, I’m trying to use FlinkML 1.2 from Java … getting this: SVM svm = new SVM() .setBlocks(env.getParallelism()) .setIterations(100) .setRegularization(0.001) .setStepsize(0.1) .setSeed(42); svm.fit(labelledTraining); The type org.apache.flink.api.scala.DataSet cannot be

Re: Flink Scheduling and FlinkML

Hello Fabio, what you describe sounds very possible, the easiest way to do it would be to save your incoming data in HDFS as you already do if I understand correctly, and then use the batch ALS algorithm [1] to create your recommendations from the static data, which you could do at regular interva

Flink Scheduling and FlinkML

Hi to all, I'm building a recommendation system to my application. I have a set of logs (that contains the user info, the hour, the button that was clicked ect...) that arrive to my Flink by kafka, then I save every log in a HDFS (HADOOP), but know I have a problem, I want to apply ML to (all) my

Re: FlinkML and DataStream API

the Flink streaming API as a way to serve your model and make predictions. > > While we don't have an integrated way to do that in FlinkML currently, I > definitely think that's possible. I know Marton Balassi has been working on > something like this for the ALS algorithm, but

Re: FlinkML and DataStream API

a way to serve your model and make predictions. > > While we don't have an integrated way to do that in FlinkML currently, I > definitely think that's possible. I know Marton Balassi has been working on > something like this for the ALS algorithm, but I can't find the

Re: FlinkML and DataStream API

Hello Mäki, I think what you would like to do is train a model using batch, and use the Flink streaming API as a way to serve your model and make predictions. While we don't have an integrated way to do that in FlinkML currently, I definitely think that's possible. I know Marton Balass

FlinkML and DataStream API

Hi, I'm wondering if there is a way to use FlinkML and make predictions continuously for test data coming from a DataStream. I know FlinkML only supports the DataSet API (batch) at the moment, but is there a way to convert a DataStream into DataSets? I'm thinking of something lik

Re: FlinkML - Fail to execute QuickStart example

Hi, No problem I'm going to create a JIRA. Regards Thomas 2016-10-17 21:34 GMT+02:00 Theodore Vasiloudis < theodoros.vasilou...@gmail.com>: > That is my bad, I must have been testing against a private branch when > writing the guide, the SVM as it stands only has a predict operation for > Vecto

Re: FlinkML - Fail to execute QuickStart example

That is my bad, I must have been testing against a private branch when writing the guide, the SVM as it stands only has a predict operation for Vector not LabeledVector. IMHO I would like to have a predict operator for LabeledVector for all predictors (that would just call the existing Vector pred

FlinkML - Fail to execute QuickStart example

Hi, Executing the following code (see QuickStart): val env = ExecutionEnvironment.getExecutionEnvironment val survival = env.readCsvFile[(String, String, String, String)]("src/main/resources/haberman.data", ",") val survivalLV = survival .map { tuple => val list = tuple.productIterator.to

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

com.esotericsoftware.kryo.serializers.MapSerializer.read(Map >>>> Serializer.java:21) >>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSeriali >>>> zer.deserialize(Kryo

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

lizer.java:556) >>> at org.apache.flink.api.java.typeutils.runtime.TupleSerializerB >>> ase.copy(TupleSerializerBase.java:75) >>> at org.apache.flink.runtime.operators.sort.NormalizedKeySorter. >>> writeToOutput(NormalizedKeySorter.java:499) >>> at org.ap

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

.java:1344) >> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger >> $ThreadBase.run(UnilateralSortMerger.java:796) >> >> >> On Wed, Aug 31, 2016 at 5:57 PM, Stefan Richter < >> s.rich...@data-artisans.com> wrote: >> >>> H

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

Richter <mailto:s.rich...@data-artisans.com>> wrote: > Hi, > > could you provide the log outputs for your job (ideally with debug logging > enabled)? > > Best, > Stefan > >> Am 31.08.2016 um 14:40 schrieb ANDREA SPINA <74...@student

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

o(UnilateralSortMerger.java:1344) > at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ > ThreadBase.run(UnilateralSortMerger.java:796) > > > On Wed, Aug 31, 2016 at 5:57 PM, Stefan Richter < > s.rich...@data-artisans.com> wrote: > >> Hi, >> >> c

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

> > could you provide the log outputs for your job (ideally with debug logging > enabled)? > > Best, > Stefan > > Am 31.08.2016 um 14:40 schrieb ANDREA SPINA <74...@studenti.unimore.it>: > > Hi everyone. > I&

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

Hi, could you provide the log outputs for your job (ideally with debug logging enabled)? Best, Stefan > Am 31.08.2016 um 14:40 schrieb ANDREA SPINA <74...@studenti.unimore.it>: > > Hi everyone. > I'm running the FlinkML ALS matrix factorization and I bumped into the

FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

Hi everyone. I'm running the FlinkML ALS matrix factorization and I bumped into the following exception: org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed. at org.apache.flink.client.program.Client.runBlocking(Client.java:38

Re: Using FlinkML algorithms in Streaming

Actually model portability and persistence is a serious limitation to practical use of FlinkML in streaming. If you know what you're doing, you can write a blunt serializer for your model, write it in a file and rebuild the model stream-side with deserialized informations. I tried it for a

Re: Using FlinkML algorithms in Streaming

Currently I am not aware of streaming learners support, you would need to implement that yourself at this point. As for streaming predictors for batch learners I have some preview code that you might like. [1] [1] https://github.com/streamline-eu/ML-Pipelines/blob/314e3d940f1f1ac7b762ba96067e13d8

Re: Using FlinkML algorithms in Streaming

Hi Márton, I want to train and get the residuals.  Thanks and Regards,Piyush Shrivastava http://webograffiti.com On Wednesday, 11 May 2016 7:19 PM, Márton Balassi wrote: Hey Piyush, Would you like to train or predict on the streaming data? Best, Marton On Wed, May 11, 2016 at 3:44 PM,

Re: Using FlinkML algorithms in Streaming

Hey Piyush, Would you like to train or predict on the streaming data? Best, Marton On Wed, May 11, 2016 at 3:44 PM, Piyush Shrivastava wrote: > Hello all, > > I want to perform linear regression using FlinkML's > MultipleLinearRegression() function on streaming data. > > This function takes a

Using FlinkML algorithms in Streaming

Hello all, I want to perform linear regression using FlinkML's MultipleLinearRegression() function on streaming data. This function takes a DataSet as an input and I cannot create a DataSet inside the MapFunction of a DataStream. How can I use this function on my DataStream?  Thanks and Regards,

Re: FlinkML 0.10.1 - Using SparseVectors with MLR does not work

hanks for reporting the problem. Because level 1 operation in FlinkML > BLAS library doesn’t support SparseVector, SparseVector is not supported > currently. I’ve filed this to JIRA [1]. > > Maybe I can send a patch to solve this in few days. > > [1]: https://issues.apache

Re: FlinkML 0.10.1 - Using SparseVectors with MLR does not work

Hi Gna, Thanks for reporting the problem. Because level 1 operation in FlinkML BLAS library doesn’t support SparseVector, SparseVector is not supported currently. I’ve filed this to JIRA [1]. Maybe I can send a patch to solve this in few days. [1]: https://issues.apache.org/jira/browse/FLINK

FlinkML 0.10.1 - Using SparseVectors with MLR does not work

All: I'm trying to use SparseVectors with FlinkML 0.10.1. It does not seem to be working. Here is a UnitTest that I created to recreate the problem: *package* com.aol.ds.arc.ml.poc.flink > *import* org.junit.After > *import* org.junit.Before > *import* org.slf4j.LoggerFac