"Read size does not match expected size" error when using HyperLogLog

Hello, I am trying to use HyperLogLog in stream-lib(https://github.com/addthis/stream-lib) in my Flink streaming job, but when I submit the job, I got the following error. My Flink version is 1.0.1. --- org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job

Big Data Interview Preparator

Hello The below is a very good book on Big Data for interview preparation. A good book on big data interview FAQ http://www.amazon.in/Big-Data-Interview-FAQs-Chinnasamy/dp/9386009188/ref=sr_1_1?ie=UTF8&qid=1459943243&sr=8-1&keywords=9789386009180 http://www.flipkart.com/big-data-interview-faqs-

Monitoring and alerting mechanisms for Flink on YARN

We are about to deploy a Flink job on YARN in production. Given that it is a long running process we want to have alerting and monitoring mechanisms in place. Any existing solutions or suggestions to implement a new one would we appreciated. Thanks!

Re: DataSet.randomSplit()

Hey all, Sorry I missed this thread. The related issue is: https://issues.apache.org/jira/browse/FLINK-2259 I checked it out then forgot about it. I'm cranking on it now. tg Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://

FoldFunction accumulator checkpointing

I'm wondering whether the accumulator value maintained by a FoldFunction is automatically checkpointed? In general, but specifically when using the WindowedStream.apply variant that takes a FoldFunction: public DataStream apply(R initialValue, FoldFunction foldFunction,

Re: RocksDB Statebackend

Is it possible to add an option to store the state in the Java HashMap and write its content to RocksDB when checkpointing? For "hot" keys that are updated very frequently such optimization would help with performance. I know that you are also working on incremental checkpoints which would also be

Re: RocksDB Statebackend

Concerning the size of RocksDB snapshots - I am wondering if RocksDB simply does not compact for a long time, thus having a lot of stale data in the snapshot. That would be especially the case, if you have a lot of changing values for the same set of keys. On Tue, Apr 12, 2016 at 6:41 PM, Aljosch

Re: RocksDB Statebackend

Hi, I'm going to try and respond to each point: 1. This seems strange, could you give some background on parallelism, number of operators with state and so on? Also, I'm assuming you are using the partitioned state abstraction, i.e. getState(), correct? 2. your observations are pretty much correc

RocksDB Statebackend

Hi everyone, my experience with RocksDBStatebackend have left me a little bit confused. Maybe you guys can confirm that my epxierence is the expected behaviour ;): I have run a "performancetest" twice, once with FsStateBackend and once RocksDBStatebackend in comparison. In this particular test th

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Sorry, I had a mistake in my example code. I thought the model would be stored as a (Option[DataSet[Factors]], Option[DataSet[Factors]]) but instead it’s stored as Option[(DataSet[Factors], DataSet[Factors])]. So the code should be val als = ALS() als.fit(input) val alsModelOpt = als.factorsOpt

Re: Powered by Flink

Thanks to you! :) On Tue, Apr 12, 2016 at 4:16 PM, Fabian Hueske wrote: > Done, thanks Stefano! > > 2016-04-12 15:38 GMT+02:00 Stefano Baghino > : > >> Hi Robert, thanks for bringing up the page. >> We're in the process of releasing our first version and would like to be >> added to the "Powere

Re: Powered by Flink

Done, thanks Stefano! 2016-04-12 15:38 GMT+02:00 Stefano Baghino : > Hi Robert, thanks for bringing up the page. > We're in the process of releasing our first version and would like to be > added to the "Powered by" page: "Radicalbit is an > OSS distribution that uses Flink

Re: Powered by Flink

Hi Robert, thanks for bringing up the page. We're in the process of releasing our first version and would like to be added to the "Powered by" page: "Radicalbit is an OSS distribution that uses Flink for Fast Data processing" On Wed, Apr 6, 2016 at 9:07 AM, Suneel Marthi wr

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Hi, those parts were examples how I had tried. I tried with your suggestions, but still no success. Additionally, there were some problems: val (userFactorsOpt, itemFactorsOpt) = als.factorsOption If I had just this, userFactorsOpt And itemFactorsOpt did not have write method. So I added ge

Re: YARN session application attempts

Hi Ufuk, sorry for taking an awful lot of time to reply but I fell behind with the ML in the last couple of weeks due to lack of time. First of all, thanks for taking the time to help me. Yes, what I was saying was that apparently from the code (and effectively as we later found out after a couple

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

Hi Kirsti, I think you attached some images to your file which show the code. Unfortunately this is not supported by the mailing list. So maybe you could resend what you’ve already tried. In order to access the ALS model, you can do the following: val als = ALS() als.fit(input) val (userFactor

Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector

How should this be done for the recommendation engine (that is ALS, example here https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/libs/ml/als.html ). I am able to run the example wit